Recording - Integrating Blackboard AI Conversations at Bryant & Stratton College.

**This transcript has been auto-generated and has not been verified for accuracy **

Scumaci, Mary Beth 0:03 We are going to get started with welcome. Chris, Carolina and Claire and I are here today to help you go through this. And we're welcoming Kelly Hamble from Bryan and Stratton College. She is a fellow Western New Yorker, a go Bills girl, I think, I hope, and right down the street from me. So I'm really excited that Kelly's here today to present. So we're just going to go through a few of our usuals, but definitely welcome to anybody who's new and just joining us for the first time. If you want to react a little bit there and show us if you're here for the very first time, that would be great. Just so we've got a vibe going there. And we're going to welcome everybody else back and thank you for joining us and taking time out of your busy days.

Likness, Eric 0:35 Not for the first time.

Scumaci, Mary Beth 0:41 So just as a reminder with the AI trustworthy framework for Anthology, that third party AI generator software is not able to be used unless you have made prior approvals with Anthology and people in the sessions. So we apologize if that's an inconvenience for anyone, but certainly a privacy and security policy. Procedure that we need to follow and about our group. If you are new here and if you just need a reminder because you join us along the way, we are all just a community. So we are just here to help each other learn, help each other share and grow from the ideas that all of our presenters are sharing with us. So we need your help. We need you guys to suggest topics we need. People to volunteer to present and that's how Kelly got on our agenda a few weeks ago. She's like, hey, I heard you've got an opening, so can I do this? I'm like, absolutely, right. We want to bring all of you guys in here and share your ideas. We learn from our peers. It's fortnightly or every other Thursday at 11:00 AM EST time or 4:00 PM BST time and. We will always post things in the community. So Claire does that for us and we thank her very much for keeping us on target with all that. And Speaking of Claire, Claire, Carol, Shelby and Chris and I helped to facilitate this lovely group. So we thank you for joining us and questions. Now Kelly's gonna take questions. She has two different spots in her presentations that. She. Would specifically like to ask questions, answer questions. But if you have questions for us, please just put a queue in front of it and then Carol, Chris and I can track those and we can keep tabs and making sure we get everybody's questions answered by Kelly as we go. So a few announcements. If you have not joined the new community, please definitely join the community, the new community and. Community that we have rolling. It's really fun getting to learn and see all the different things that are happening and try and find them too. If you guys are like me, I've been looking for a couple things, but we do find them. So Chris just posted that in the chat for us, so please do link in there and we certainly want you to focus on becoming a member of the Blackboard user group if you have not done that already. So do look for us. On the user group section of that new community, we want to make sure that everybody is very aware of Digital Teaching Symposium that is going to be on November 13th this year. So as always, there are going to be a whole lot of great presentations and Chris has that posted for you, the link in the chat as well. We also want to make sure everybody's aware that on Thursday, oh, I'm sorry, Claire, we're the 13th, right? We've got the 11th on this slide. I think we made a typo here that when the day of the digital teaching symposium, which is November 13th, we will not be holding this Blackboard user group so that everybody can attend the other sessions.

3:09 Yes.

Sandra Heidecker 3:10 Thank you.

Scumaci, Mary Beth 3:18 That are going to be running at the same time. So sometimes we have a gap where we can run the group where there are no other sessions going. That's not happening this time. So we want to make sure that everyone has a chance to go to a session. So we will not be holding our Blackboard user group for that week. So just mark your calendars for that please. And we have your Fix Your Content Day coming up. So that is on Tuesday, November 18th. So if your institution is interested in helping to play and fix your content day, go ahead and register and sign up and let's see who comes out at the top. I think it was Shelby School who won, if I'm not mistaken last year. So that is coming and we're also going to do a little commercial. Here our next session on October 16th, Fiona Stewart from the University of Sheffield is going to be talking about the TDM course catalog and doing a presentation there. So bring your ideas. We're using it here at our institution. We haven't been using it that long, so I'm curious to hear what Fiona has to say as well as the rest of you and the experiences. That you've got to bring to the table. And now we have for our very special presentation today, we have Kelly Hamble, a course developer from Bryant and Stratton College, and she's gonna be talking about integrating Blackboard AI conversations at their institutions. So Kelly, I'm gonna stop the share here and let you introduce yourself and pull up that presentation so we can get going.

Hambel, Kelly E 4:36 Sorry, let me unmute myself before I start talking and I'm just talking to myself. And while I get the screen set up, yes, I am in the Buffalo area, but I will say I'm a little bit of a Sinner because I did not grow up in the Buffalo area, so I am not a Bills fan. So I apologize to everyone on the call who might be thinking that. So, oh, just one second. Alrighty. So I want to thank you all for being here. I know we are coming from all over today to talk about this fun and exciting Blackboard A I conversations, especially how we integrated them into our course builds here at Bryan and Stratton. Before we start though, I do want to introduce you to my friend that you see there on the slide. That is Bernie the blue bot. He has become our design team's little mascot for integrating A I, using it and really having fun along the way. So hopefully he can have fun with us today as well. I will give you all a fair warning that since we are now in October. And Halloween is quickly approaching, which is his favorite holiday. He'll be dressing up along the way as well. So to begin with, these are going to be like the points that I'm looking at. So you all can get an idea of how we approach the A I conversations and how we're looking at them moving forward. And oh, sorry, I don't really want to have that breaking on that slide that was supposed to be removed. I don't think we really want to be telling Blackboard or Anthology that we're trying to break their fun new things. Breaking was certainly not a part of the process at all in any way. Really was. But anyways, just as a refresher in case there's some of you who haven't really used AI conversations. This is a reminder that we do have the Socratic questioning and the role play, and both give really interesting ways for students to look at topics differently. And before we dive in too far into this presentation, I do want to take a moment and ask you all to pop in the chat which one you prefer. I know we already got some. I believe it was Bevin that was saying she liked the role play. Persona better. But is there any other people that are like, yes, I love this one. The other not so much. Let me see. Let me get my chat up. Role play. We like the role play. I will say this credit question has its place.

Sandra Heidecker 6:39 OK.

Hambel, Kelly E 6:51 I think for the right course, it's amazing. We haven't got there yet at Bryan and Stratton, but yes, OK. So overwhelmingly role play. Great. I think that's totally fine. I think it's the one that's a little bit more malleable. I will also say the anthology and that Ava that's coming up, we've been testing that out that. Might take over as my favorite Blackboard AI feature. We got to test it out a little bit more and see how we feel, but we'll see how it goes. OK, so first, to give you an idea of who Bryan Stratton is as a college, I think it really helps understand. Our process that we took to integrating AI into our courses. So we are a career college that prepares students for the workplace. We have several campuses across four states, New York, Virginia, Ohio and Wisconsin. And we also have our large online campus that is based in Buffalo, NY. Despite we sorry, despite having these five campuses, we do take A1 college approach, which is A1 content model that helps students meet all of the learning outcomes regardless of the modality that they're taking the course in. So whether it's a fully online program or they're taking it face to face or somewhere in between. To support the one content model and one college approach, we do use course templates that are used on both of our campus courses and for our online as well. So the students, no matter how they're taking that course, they're still getting the same core course material and a quick summary of that build 'cause I know we are really unique. Not a lot of schools use course templates. A lot of schools still have faculty build the basis of their course. Our build process does involve a curriculum manager and our design team member, which are instructional designers working together with a subject matter expert or SME to write, build and design the course. The curriculum manager oversees the creation of the assessments and ensure that all of our course outcomes are met. Once the assignments are complete, then the SME then works with the designer to create all the weekly material, which it includes but not limited to our lectures, our discussions, and our weekly descriptions. Once we have all of that created, it then does go to a committee of experts in the field that review all the course material, make sure that it's accurate, relevant, and again, that all of those course outcomes and program outcomes are being met. Brian and Stratton also just got a new strategic plan this year, and one of those goals is to leverage AI efficiently so we can prepare students for their future workplace because we know it's out there and it's going to be essential moving forward. By incorporating AI, we're also giving our students a competitive edge. When our design team learned that this was something that the college was really pushing for and really supporting, we got an idea of how we could start looking into building this into our course builds. And this is how we got started on our journey was having the AI committee, which I'm currently leading. And it's four of us that have a background of working with AI, really understand how to use it. And we're training the rest of our instructional team, which total right now is 17 of us because that includes our design team and our. And our curriculum managers, so that we're all on the same page and we're ready to work with SMEs to use AI in their course builds on the committee. We do say that we are the AI exploration committee because we want to go out there, you know, explore new tools and programs and see if it is something that we actually can benefit from. And then we bring all that information back and all those skills that we've learned along the way and share it with the team. That's why Bernie right now is dressed up as a little explorer, because essentially, like Bernie, we're venturing out into the growing wilderness that is a I tools and features and then sharing what we've learned once we return. Obviously, one of our main task as the AA committee was to really ensure that our entire team had the same understanding of what AI was and how to use it. So what is a prompt? What are the limitations? What programs are better for certain actions? Because I'm sure, as we've all seen. There's a lot of hype around certain A I programs or tools, but they only work for very specific things in very specific ways. We also wanted to make sure that our team understood the human will always be involved and it's OK to look at something that you're given, so the output from the A I program and go. Actually, yeah, I did spend some time working to get this, but this is not what I want. I don't want to use this. I would say that this last point was really essential because like we see across anthology, A I needs that human. We want to make sure that our team is always keeping that in mind. The human is always there and it's OK to understand that it's not what we want, so we don't use it. And then once we had, oops, sorry. And then to ensure that we had the same understanding of where everybody was. To get started exactly with our trainings, we sent out this survey because all of these questions gave us an idea of where everyone was at and what they understood. So I'll give you a moment to just read those over. So from these questions, it really gave us an understanding of where the team was at a whole as a whole. We didn't want to start with the Super basics. If everyone was already like, yeah, I'm using it on my daily life, I know how to use it. Or we didn't want to start way too high just because our committee had that more advanced understanding of it. Overall, I would say that the answers were pretty much what we expected. We did have a few people that were not on our committee that said, yeah, I've been using it, I really like it, I understand how to use it, how to prompt, engineer, etcetera. But we did have over half of the team say that they had very limited experience and shared that. You know, I know what A I is, but that doesn't mean I really know how to use it and I haven't really tried to use it either, which this is completely fine. I think for a lot of people, unless you have some general interest or some invested interest such as I know this will help in my day-to-day life, it's really hard to turn around and try to. Investigate a new product and spend that time to get there where you're at a good place. Like, why try new things if you're really not sold on it to begin with? Also in this survey for questions, like you see, we asked what are the concerns and our team members did voice concerns about using a I. And like I said, all of this is valid. It's still new, we're still learning about it. So it's normal to have these fears and concerns. Especially when we see countless. Upon count, I feel like it's every day there's a new story about, oh, there's this with a I that happening that make it seem like you don't really want to use it. And so to summarize what those main fears were, the big one and the main fear that we do see on the news all the time is going to be that job placement. Because our team felt, you know, if we're using these A I tools and programs to do these jobs, then what's the point of us? What is our role? There's always, always a new story pretty recently, like maybe not every day, but I would say. Quite, quite every few days of how some company was like, yeah, we just laid off this entire department and we replaced them all with AI and we're saving so much money and everything's so much better. So they say. Another concern that our team had was the loss of creativity and human touch scenes. A big part of our job is creating fun and engaging interactions for the students so that they're learning in a fun manner and not just going through and reading. And you know, if we're having the AI do that, then maybe the courses are going to start to feel very sterile and lifeless. We're not going to be able to have that fun element. In there. The very obvious, the ethical concerns and biases, because these programs are trained on people. Unfortunately, people are very biased and not very ethical all the time. And I'm sure we've all seen where certain programs are biased towards certain races or certain genders, or they only depict certain people in certain roles, say. I'm trying to generate an image and I say, can you please create an image of an middle-aged Dr. It's most likely going to be a white male. And we don't want this to be what our students are seeing because we have a very diverse student body. We want what we're showing them to reflect not only the world around them, but also. Reflect them so they can see themselves in their future careers. There is also the fear of an over reliance on AI. If you're using it every day, you're obviously going to forget how to do things because you're not the one doing them anymore. And we felt that if we're over relying on it, then we're also going to lose sight of our goals, like we're not going to be focusing. On having nice and engaging courses that students really enjoy. And the last point that sums up a lot of the issues that people shared was the lack of critical thinking, not only on our side, but also on the side of faculty and students. So our team said that, you know, if faculty is using AI, they could be giving crappy feedback, not giving good, solid, constructive feedback that students can use to help learn and grow. Students, if they're using it too much and relying on it, they're not really learning either because they're just plugging. Lugging one thing from one AI into another AI to get their answer and going on with their lives. And then obviously critical thinking on our side because we're not thinking through how to make these courses as interesting and as engaging as possible. I'm sure there's also a lot more fears, but I do want to pause for a second and I want to ask everyone, are these the same type of fears that you're having at your institution? Is there something else? I don't know. Share. I want to see how everyone else is dealing with it. Pop it in the chat, please. So yes, the same fears. Anyone else? About fears. No. Yes, exactly, Marybeth. That's a great point. Fear of the unknown. Because like I said, this is still something very new. What does it do? How are we going to use it? Yes, data privacy. You sound like our IT because our IT department is constantly saying data privacy. Don't put all the student information in there. Not that our team even has access to student information, but they are telling us that very often. Yes, intellectual property, privacy. Yes, great. Thank you. So as you can see, there's a lot of fears that are in line with this, but even fears beyond this. And I think all of these fears, the ones that I have on the slide and I shared with you all and what you're putting in the chat. Yeah, I'm paying for this degree and it's made by a I I think these are all 100% valid concerns. And so this is something that the committee kept in mind as we began our training. And so our first step to show everyone that, you know, your fears are valid, but also this is the reality. Was to show them what we plan to do with the AI tools in our course builds. So for one, you're probably already using a little bit of AI in your life, whether you know it or not. For example, if you use Siri, Alexa or Google Home, you're using AI and you might be a little dependent on it. I know sometimes, especially when. I don't feel like typing out my whole question to Google. I hit the little mic and I just say what my question is and I get my answer and I'm like, great, thanks and I go on with my life. Now that is non generative AI, but it's still something that's in our lives. We also reassured the team that. We want everyone to look at a I as a thinking partner, whether we're talking about how students are going to use it in our courses, we're talking about faculty using it, or our team while we're building the courses themselves. This thinking partner, I will have to say I did steal this phrase from someone on our team because she said it and I love it. But think of the A I as someone to be there to give feedback to what you're saying. A sounding board, or maybe a more interactive and more comprehensive Google search. You know, something to help you work through your ideas. It's not a do it for you tool. The next point I will say comes with a major caveat, so I know I did keep it on there cause I was like it needs to be there. So in theory it can enhance efficiency if I can say that word today in speed on certain task at certain times when you're using the right program. So that's the major caveat. We did share with the team that we look at it as once we develop real skills with the right AI for the right action, it will save us time. But also understand that it might not always save us time, especially if we're going in and we have to make a bunch of prompt iterations. Because the output it just not there, not like getting there, but not what we wanted. Um We also shared with the team that this would also give us the opportunity to give students much more personalized learning experiences because the courses are going to be shaped by what the students are saying to them. The AI. So they're not all getting the same exact interactions and the same exact answers. And most importantly, like I've already said in this presentation already, the human will remain. We do not have any plans of saying this is how we're going to replace half of our team with AI. We just want to look at it as something. As a tool to help us in our daily lives. So once we really worked through with our team and reassured them what it is, then we got to talking about the basics. So what is a I that generative versus non-generative? How to write prompts because engineering prompts is an art form in itself. Whether it's writing the initial prompt or even editing it for your iterations, obviously got into the ethics of it. We're not putting any of our Social Security numbers or any of that into them. And again, like I said, we really stress that this is a learning tool to help us, not a do it for us tool. And that's something that we need to keep in mind throughout the process. Once we had ensured that everybody was at a good comfort level, we started diving into Blackboard AI. And I'm sure like a lot of you, obviously we were very interested in the personas because it created a way for students to interact with some. Something in a more engaging way than if we just used a regular assignment or discussion post. But knowing in general what Blackboard AI conversations is and having a solid example are obviously two different things. Here's where Sarah, our committee member, stepped in because at the time. I'm actually her SME was working on a human service services course and had mentioned that she wanted to make interactive scenarios for the students to have a more immersive experience. And so her SME went through and created these real-world feeling scenarios that would give the students the experience of working with patients as, say, a caseworker, therapist, or human service professionals. When Sarah was talking to her SME about this, she's like, this is a great opportunity for AI conversations. Would you like to do this? Because we do have scenarios in our courses already using programs such as Rise or Storyline, but we saw the potential of using this new and fancy tool to make that even better and take. To the next step. So that's how Sarah in her course really became the A I committee's Guinea pig. And like I said, we did have a handle on how it worked. We had tested it out, but there was no solid plan at this point of how we were going to be putting it into a course and making sure that it worked well. Obviously we didn't want to take a new tool, just throw it into a course and tell students and faculty, good luck, you'll figure it out. We wanted to have something that we had fully vetted, ensured that it fit within our course guidelines, so. We really went into this with general suggestions and general guidelines of just saying, you know, we've tested it on these types of scenarios. These are the results we got. So these are the things that like a checklist of what we think you need to have. So Sarah put Hearst me into a sandbox, made sure that they had all of the AI permissions. that they needed and then went through and really explained everything that we had known at that point. She also shared again what we thought at that point was best practices for setting up the student directions and the details of the persona, but that has thankfully been refined and then gave a really rough template of this is what you need. These points are being met. Once the SME had written it and previewed the chat to make sure that it was working on her side, Sarah actually copied the conversation into our course templates to keep building out the course and also for us to start testing and tweaking the conversation as needed to ensure that it was ready for the students to. Use in our September term and if you do use SMEs and your course builds like we do, this was a really important point to move it out of the sandbox or at least cut off SME's ability to edit it because we moved it out of the sandbox because we wanted to make sure that when. We're changing things and saying, OK, we're having this problem. We changed X and now it's working. We didn't have the SME going in the next day and making more changes because that obviously would make more problems. So after Sarah and the SME had finalized everything, it was time to break the personas, I mean. Obviously, test them out. Sorry, this is an old slide that I clearly didn't update, and I did tell Bernie that we changed the slide and he needed to use the hammer. But you know how these bots can be. They don't always listen and do what they're told. So anyways, sorry, let me get back on track. So the thinking behind our testing was to look beyond the general questions that we had for SMEs when they were building a scenario of OK, can the students follow it? Do they understand their objective? Instead, we had the major question of how would how would students experience this? And how would all types of students experience this? I'm sure we can all agree not all students are the same. As much as we would love to assume every student that walks through our door, takes our classes, loves every minute of it, enjoys doing the work, there's always a range. You do have those students that are really into it. They love the course, they love the assignments. They will argue with you when you give them a ninety-eight out of 100 instead of 100 out of 100. But you also have the students that know C's get degrees, and they very much live by this motto throughout their studies. You got the students that are a little bit in the middle, like do do about average work. But then you have my personal absolute favorite, those students that just say, oh, you know, I'll just ask for extra credit at the end of the semester. I'm not gonna bother doing any of the actual assignments. So knowing this range of students, these were what we had in mind when we were trying to break, I mean test our personas. We did detail in the student directions that when the student is interacting with the persona, they need to start the conversation with a. Reading state their role and then progress from there. But like I said, are all students going to be working that way with the AI persona for various reasons? So that's why in our testing we focused more on the student that maybe didn't even read the directions at all beyond talk to this. AI persona. Um And obviously I will say for a moment before you all start to think that we only tested it for bad things. We did make sure if the student did exactly what they were supposed to follow the directions like they are supposed to, the AI did work. Right. But like I said, our goal was to make sure that it worked for everyone, for all students, however that might be. So here on the side, I did give a little preview of what all these three personas are. Obviously there's a lot more to their directions and their descriptions. But that's obviously a lot to show on just one slide, let alone on three sides. So here's just the gist so you can understand some of our testing and some of the results that we received while we were breaking the personas. The first person that we're going to take a look at is Jasmine Walker, who's been meeting with her case. Worker every week for about the last six months. Jasmine is a very warm patient, but she does get stressed and overwhelmed. So that's why the student's goal is to check in, cancel the appointment for this week, but make sure they're doing it in an empathetic manner. And when we are the disengaged student just saying hi, we can quickly see that this becomes a less than ideal interaction. So I'll pause for a moment so you all can read what Jasmine said. So as you can see with this interaction with Jasmine, I don't think she really knows who she is. Even when the student is trying to correct Jasmine, there's still obviously a little bit of miscommunication. On Jasmine's side, all on her own. So even though that the student is trying to kind of correct, we still even see Jasmine. OK, I am the patient, but I'm sorry to hear about canceling our session. Well, where did the student say anything about the session, let alone canceling it? So she's still jumping ahead with information that technically she shouldn't know, but because she was in that wrong role. So the fixes that we made to fix this persona was going through in the directions and really stressing the role of the student and the persona in the directions and also in the persona description. We also shifted the persona description to be more of a paragraph about Jasmine, because before we had bullet points detailing the different characteristics of her. For example, we had one like communication style and says she's warm, likes to talk over text. By just changing that into one solid paragraph and removing those bullet points, we saw that that helped Jasmine really know who she was. And that helped it where we can say, OK, if we're being a little bit of the disengaged student, we can still go in and talk to Jasmine and get the experience that we're supposed to get. As this fictional caseworker. And with our next persona, we saw another issue, but a different one. So just as a review, our persona for our other one is going to be Tina Smith, who's our caseworker. She has a lot of experience in the field, but is currently really overwhelmed because she has a very high caseload. She doesn't have a lot of time to talk. But she would love to set up something to discuss a plan for Sylvia, which is the patient that the student is checking on. In this scenario, the student's role is a Human Services professional checking on her high school student, her high school student patient, Sylvia. This is the first time, keep this in mind, the very first time that the student is reaching out to Tina, and they're going to try to set up a Teams call so that they can discuss Sylvia in more detail and make a solid plan to move forward. Let's see how Tina responds. Sorry, I was taking a look at the chat real quick. So as you all can see, we don't have the issue of Tina going like, well, who am I like Jasmine? But instead she's rushing ahead and dropping all of this information about Sylvia when the student just says hi. This is an issue because in order to have a realistic conversation, there needs to be more back and forth, especially since this is the first time that the student is supposed to be interacting with Tina, and the goal of the interaction is to build up that relationship. And we even see that Tina mentions in that second message. Oh yeah, your previous message, but it is the first message. It is the second on the screen, but it is the first message that Tina is sending to the student. To fix this, we included Tina's name in the description as well and made sure that we defined it in the. The student directions. So it was very clear that Tina is slow to answer. She's not dropping all this information. We also stressed that it is the first interaction that relationship needs to be built up. We also told Tina that she needs to respond in shorter messages. Come on, give the sense that you are busy. If we're busy and we're trying to respond real quick to something, we're not always given a full paragraph. We also found that lowering the complexity of her responses also aided immensely into this info dumping. Then once these changes were made, Tina went back to exactly what we needed. Like, hey, OK, So what can I help you with today? What's the issue? Because that's what the students needed to experience. And our last one is our Jordan Myers. He is our 31 year old man who's in therapy for his anxiety. And alcohol use. And he's our persona that's supposed to be defensive because he's not doing what he's supposed to and he doesn't want to admit it. Last night he was supposed to go to a support group, but we know that he didn't because the student in their role as a therapist is checking in on him after the group leader had shared that. Jordan missed that very important meeting last night, so let's see how Jordan responds. Sorry, I'm trying to make sure I give everybody enough time to read it without giving like too long of an awkward pause. So as you can see, like Jasmine, Jordan doesn't know who he is either. And if anything, we're a little bit in a loop where he's. Rather insistent that he is the therapist and he's checking in on you. The fixes we made for Jordan are going to be along the same lines as Jasmine, but we actually had something different with an issue with him is actually his entire persona was written in passive voice. We didn't think this was an issue. I don't think we really noticed it until we were seeing these issues and we saw that making his description all in the active voice helped a lot of the issues of him being very adamant that he was the therapist. Again, we also, like I said, we defined the roles, made sure we had names in there to help the AI move along. And then once that was all done, when we talked to Jordan, he was quickly defensive. Like you said, hi, how are you? Yeah, why are you? Why are you talking to me? Like what's what's wrong? I'm I did what I was supposed to, which is exactly the experience that we wanted. The students to have. So once we had these personas set up, we had tested them out. As you can see, did in fact actually mildly break them so we could know all the potential issues. We were ready to start sharing with our faculty all about this new feature, especially if they were going to see it in their courses in our. Our September term or our upcoming November term? So at Bryan and Stratton, we have the Academic Learning Community or simply the ALC, which is the place that provides professional development for faculty. Our design team has led a lot of training through the ALC recently because we just completed our conversion to Ultra back in May. So it was the natural next step that the AI committee would use the ALC to make sure that the faculty was prepared and fully trained to use this new feature in their courses. While we did focus on the faculty that would be having it in their courses either September or November, we did actually also invite all faculty who was just interested about. Learning more of what the design team was doing with AI and how it was being put in part of our courses. So in this presentation we shared an overview of what the AI conversation was, detailing what the Socratic questioning was versus the role play and explaining why this was actually. really an enhancement to their courses. Just like we shared with our own team when we were training them, we made sure that we gave them the faculty space to voice their concerns, ask questions, and then reassure them of what we learned about the AI. Obviously, we don't plan to use it to replace any faculty. That was a major concern. But along the same lines, we also assured them that this was indeed an enhancement to their course. Students are on at all times of the day and night, and this would be an activity where students are getting instant responses no matter what time they're interacting with the A I. It also was a way to give students low pressure class participation so that they can talk about their ideas, really develop those ideas in a way that maybe they're not comfortable with doing in a regular traditional discussion board. I think we've all been there. If you've ever done any type of course online, you write. Post on the discussion board and you're like, oh, everyone's reading this. I misspelled that word. I also misspelled that word. Did I say that phrase right? Oh, and then you're stressing about it, where when it's interacting with the A, even if you do spell things wrong, it still understands you. It still moves the conversation along. And so students don't have that anxiety that could come from a traditional discussion board. And we also shared with faculty that it is a way to start giving students a more personalized experience because the A I is responding based off what the student is saying to them. We also, most importantly, preface this tool by stressing no AI tool is perfect and we are very much aware of that. When we were testing slash breaking these personas, we did find certain issues. To try to compensate that, because again, it's not perfect, we do see a little bit of the role reversal still. If it's the right way, which we hope students aren't doing, we did add a little disclaimer in the direction saying if this happens, simply restart the conversation. But again, we know that students don't always read directions, so we shared this and made sure that faculty understood there are certain known issues and this is what they should do, especially if the student doesn't read directions and reaches out to them with a concern or an issue with the A I conversation. We told them this is all normal depending on how that student is starting their interaction. Additionally, we reassured them that our design team is always here to help if problems happen, if they want to discuss it more, or just get a better understanding about it. While there are known issues, we really encourage them to share all of the. Problems that they have because if it is something that we don't know, it helps us learn more about the product and then we can also share those experiences with anthology and help refine the tool even further. Currently we do have these assignments as non graded in our courses. We don't plan to have this forever, but we're currently doing this to kind of ensure that there's a smooth transition for students and faculty as they start using it in their courses. We know that AI is still new and something that not everybody is excited about for various reasons. Additionally, there's always a learning curve when you're using something new. New, as we saw with our breaking of the personas, it's also not perfect. No AI tool out there is perfect and works 100% of the time as we expect it. Therefore, we wanted to accommodate for this by making the assignments not graded at the time so we can confidently say, you know, if students have an issue, at least they're not stressing. about their grade along the way. And just because the AI conversations are ungraded, we also needed to make sure that we trained faculty to still go through and review it and give constructive feedback to help with the students with their conversations. Because without having that human factor, the students aren't getting the full experience. Erience that we exect them to have with the AI ersonas. And as with any new adventure or experience, we learned a lot about the A I conversations through building these Human Services personas, definitely refined our best practices. And these are the main points that I would say that we learned that I hope maybe would help you all as well. So the first point I think on the surface does look very self-explanatory, like of course you have to be specific. But I would say that we actually had to be more specific than we expected we would have to be in these scenarios because we didn't want the students just to have a general conversation. We wanted them to have. A specific role, such as a caseworker doing a specific action, and to do that you needed to make sure that you had those fine details on both sides, so on the student directions and for the persona without overloading either side as well. So it's a fine balance to have. Also, like I said with our first example, Jasmine, the bullet points, I felt like this was the most shocking for me personally because you see so much where the output from the AIS and bullet points. So it's kind of, I don't know, at least for me, a natural like, OK, I put bullet points. But that actually led to more confusion for the persona, and it was better to have a whole paragraph that was a narrative of who the person was, and we saw that it led to much less issues. We also learned when we were talking to the development team of the A I with anthology, that using active language made all the difference. For example, even just saying Jasmine is in therapy could cause some issues where if you say Jasmine receives therapy is a better way to have that persona description. On the same note, we were also told that stressing aspects needed to be very firm. So you don't just say Jordan must not ask about the therapist, Jordan must not in all caps for that not ask about the therapist. And we did see that this helped fix a lot of our issues. Also, we cannot assume like I've been saying throughout this, that any AI tool is perfect in how in how it works or how a student will. Interact with it. They're not always going to be doing what we want them to do. As we saw in these examples, if we're a little bit of the student that's not engaged, just saying hi instead of saying hello Jasmine, how are you doing? And all of that, we don't get the correct experience. We even. Had this with the Socratic questioning. While our focus had been on role-playing, we had tested out and trained the team how to use the Socratic questioning and even that we were able to break it. In one of our training sessions, our team member was using the Socratic questioning. And she very much felt like she was falling into this endless loop of conversation. Cause it's like she, as she said, why does it keep asking me questions? But I will say that we did talk to Richard Gibbons about that and with a lot of the new updates that they're bringing, that shouldn't be a problem moving forward. So just FYI, not to further dissuade anybody from Socratic. Questioning since I know the role play was the more popular one in the chat, but additionally we sorry. So that's why we thought that multiple attempts were important is because if it's not acting right, we need to be able to restart that conversation. And give the students another try. We don't want to discourage them, especially if this is their first time using AI. A lot of our student body is older, so we also can't assume that they know it either. And overall, I would say that a lot of the changes that we had to make while we were going in and. And changing our personas might seem like common sense, but they're really not, especially if you've done any deep diving into prompt engineering for programs such as ChatGPT and Copilot. To set up the personas, you have to speak differently than when you're interacting with those, and sometimes really striking that. Solid balance of giving the AI persona specific information that it knows who it is, knows who the student's supposed to be without overloading it. It's a fine balance, and it's not a balance that you can say this worked for Jasmine, so this worked for Tina. It really does differ from scenario to scenario. And from these different points that we learned, we also have changed how we're looking at AI as a whole, because we want it to look at it as something that we're using meaningfully and not frivolously. We want to make sure that it's something that is going beyond our traditional assignments and we're. We're not just putting it in courses because it's new and shiny and everyone's doing it. So I do have actually a bonus conversation to share with you all that I think reflects our thinking moving forward. So we have real quick two course career development courses. One's a general one, getting students prepared for the workforce. Where career development too is for students who are already in their career and say they're coming back to get a degree so that they can move on and get promoted and stuff like that. In both of these courses, the goal is to have them do an interview, see what it's like. However, when we were rebuilding this course, Arsami mentioned that. Really, if you're having a promotion, which is what Career Development 2 kind of focused on a little bit more with its interview. If you're having a promotional interview, it's not always more comfortable. Some companies do have it where you need to do an interview to be promoted. Since you're already in the company, since you might know the people, it could be maybe awkward. So this is where our friend Carl came in. So there's his little blurb and so. We wanted this scenario to really reflect our goal as well going forward because it's giving students an experience that is really almost impossible in an online course and rather difficult even if you're in a face-to-face classroom. And I think that's what really makes it so valuable. So yes, an instructor could conduct a mocked interview with their students in their classroom, but if they try to be a little bit on the colder side, it could be awkward. They can't do it as well as Carl can, because it's not something that's always easy to maintain. Especially when I think all of us who's ever taught, you really try to make your classroom a place where students feel comfortable. You try to make it warm and welcoming and inviting. So you take, you spend all this time building a nice classroom. You don't really want to mess that up just to make the students feel uncomfortable. So they get a real, more, a more. Realistic view of what an interview can be. But with our AI persona, that's fine because it's not you being a little cold and standoffish. It's Carl. He's the one being distant. So as we'll see in the moment, Carl is not rude. But he is definitely focused on getting the information that he needs and nothing else. He makes it clear that he has no time for small talk, and he keeps moving the conversation along without trying to give any praise or any smooth transition. And once he asks all of his questions, we have him say, OK, we'll contact you later. So he is impersonal and distant, but he is respectful throughout. And I think we can all say it's not the best interview, not the worst, but definitely not the one that you want of you come in and someone asks you, hello, how are you doing? How are things going today? So here I'll go ahead and give you a moment to look at what Carl said. So obviously it's not an awful interview, but it's not the most friendly for this. The students are. They can either use their actual career, say like I'm at this company, I want to be promoted to this one, or they can use something as Google. And they're trying to be promoted. You can see that this is a bit of a cold interaction. He asks his question and he goes on. And even in a regular interview, I think this is a valuable experience because most interviews are not fun. I don't know about you all, but I've definitely had quite a few experiences. You walk out of the interview and you're like, oh, I never want to do this. This again, I'm happy it's over. So Carl gives this valuable experience that, like I said, is difficult in the classroom. But by leveraging AI, we're able to give this very valuable experience nonetheless, while I do think all of the examples that I shared today already fit our idea of having thoughtful. Our mission moving forward is to make sure that the SME is very much aware of that, and if what they give us as a scenario doesn't fit within it, we're not going to use it because we want to make sure that we're using it in a meaningful way and giving students an authentic experience. As I said, our mission is to leverage AI to truly enhance our students learning experiences and nothing more. And with AI conversations, we've started on a great path of doing that. So Bernie and I would like to thank you all for letting us share about our experience. I did also have to sneak in my own little fur baby pebbles. Say hello to all of you, and because I clearly used to teach, I do have a little handout that I made that I'll share with you in chat in just a moment. But if you have any questions, let me know. Otherwise, I hope you all enjoy the rest of your day. And I'll get that to O in chat.

Chris Boon (LLE - Staff) 51:39 That's wonderful. Thank you so much, Kelly. There's been so much interaction and engagement in the chat. As you can see, there's lots of love for your fabulous little character in the graphics, so much discussion about these scenarios. Absolutely brilliant. Just in case you hadn't seen Mary Beth's having some campus technology problems.

Hambel, Kelly E 51:59 Oh, no.

Chris Boon (LLE - Staff) 51:59 Today, so I'm not sure if you can get back in the call. So I'm just sort of jumping in and helping out at the end here. We have got a couple of minutes for questions. I think there's been a lot of discussions in the chat, but I did notice earlier.

Hambel, Kelly E 52:08 Why it's not letting me?

Chris Boon (LLE - Staff) 52:15 Jennifer asked, would you be willing to share the prompts for your personas? I think that would be really interesting because it's been so, so good to sort of see the results and the the glitches you you've that have come to light. But it would also be interesting to see sort of exactly the the description.

Hambel, Kelly E 52:22 Yeah.

Chris Boon (LLE - Staff) 52:34 Of the personas you've entered into the system. And also I think you've mentioned about the some of the nuance of the describing the persona and the the way you've used a different voice and even just the emphasis using the capitals to emphasize really important things. I think I don't know if you'd be willing to sort of share any sort of specific examples I think. We could really all learn from some of those. You know, if there's a situation where you've added, you've described something in one way and it didn't work, you describe something in another way. I think those sort of subtleties would be amazing.

Hambel, Kelly E 53:07 So we actually have a very long document. Sorry, cause like I am listening, but I'm trying to find cause I'm like I just had. Oh here it is. I'm like I just had the file. I made sure I pulled it up so I could. And now of course it's not letting me put it in the chat. Oh jeez.

Chris Boon (LLE - Staff) 53:18 Amazing. Don't worry about sharing things at the moment because we have had a request if you would be willing to share your slides again because there's so much fabulous information.

Hambel, Kelly E 53:22 OK, OK. Yes, yes, I'll share my slides and then I'll share that there because I don't know, Teams is not letting me do anything right now. So yes, I would say it was definitely an experience. I would say the biggest thing that we saw would be when we first started testing it out as a committee because we.

Chris Boon (LLE - Staff) 53:32 That's great. Yeah.

Hambel, Kelly E 53:46 We went in with like bare bone scenarios and we're like, OK, this and we did see that it was pulling weird information. What was the one? Because there was one that we did that we actually took from a course that we already had. And it did not mesh well in the A I persona at all. And I am completely blanking because I stressed way too much about doing this presentation. And now I'm just like, I forgot everything else.

Chris Boon (LLE - Staff) 54:17 Oh, no, that's fine. I mean, we're more than happy to. So if you're willing to share things in the handouts and things afterwards, that's absolutely fine. And please, please don't stress. We thoroughly appreciate the effort that's gone into this. You've clearly prepared for this so much. And yet some of the comments I see Vicki said outstanding presentation. Yeah, the the amount of feedback.

Hambel, Kelly E 54:25 Yes. Thank you.

Chris Boon (LLE - Staff) 54:36 The the positive comments here is wonderful.

Hambel, Kelly E 54:40 Yeah, I'll I'll get everything together and try to have something that at least makes sense to everyone else, cause I think it makes sense to us since we were the ones like going in and playing around with it. But we did notice that sometimes even just a single word actually made a difference as.

Chris Boon (LLE - Staff) 54:50 Yeah.

Hambel, Kelly E 54:58 I think mildly silly as it is, like she receives therapy rather is in therapy. That changed it where, yeah, we still had the role reversal at first, but it would be slower when it happened in the conversation, which was a strange, I think, thing to see. Like I said in the presentation, it's definitely something you can't go into it and be like, oh, this is how I talk to ChatGPT, so this is how it should be. It's it's definitely different. You have to try to choose your words carefully while when you're interacting with it, it's fine.

Chris Boon (LLE - Staff) 55:29 Yeah. OK.

Hambel, Kelly E 55:34 If you misspell things, you say the wrong word, it doesn't care. But definitely having the persona and saying who it is in creating that full background made all the difference rather than just saying it's a patient with this issue and they're talking to their therapist. Well, what else about them? Because that helped bring in information as well in the conversation.

Chris Boon (LLE - Staff) 55:56 And I think as you said, it's prompt engineering, isn't it? It's and the subtleties with that prompt engineering makes such a difference. I think you've really sort of hit on a nuance that I know haven't come across before, the subtlety of that prompt engineering. I think we've seen some great examples. I always go back to Rob Farmer from Northampton's examples where he spent a lot of.

Hambel, Kelly E 56:02 Yeah.

Chris Boon (LLE - Staff) 56:16 Time sort of going through different scenarios with with the A I personas, but I think you've definitely hit on a different angle. I I definitely haven't seen before. So that's been absolutely amazing. I'm just aware of the time. It's 2 minutes until the top of the hour. So if anyone's got any very quick questions, anyone want to put a hand up and ask anything that's not been covered.

Hambel, Kelly E 56:25 Yeah.

Chris Boon (LLE - Staff) 56:35 And Carol, I don't know if there's anything sort of major in the term in the chat that we've missed out on. I've been trying to keep an eye on things now.

Carol Chatten 56:41 I don't think so. Apologies if so, but no, I think it's basically just praise for what a fantastic and useful presentation.

Chris Boon (LLE - Staff) 56:50 Yeah, fantastic. In which case, I will just say thank you so much, Kelly. You've clearly put so much effort into this. We really, really, on behalf of the community, appreciate you sharing all of this information. I think everyone's going to look forward to seeing the handouts just to go through everything again, but absolutely wonderful.

Hambel, Kelly E 56:57 Thank you.

Chris Boon (LLE - Staff) 57:10 So if I can just let's have a go at presenting the last slides. Go on PowerPoint live. We don't want that one we want. So it is just basically to say thank you so much. I wish I could just go straight through to the the right slide. We will get there eventually. So don't forget we are missing the session on the 13th of November because it clashes with the Digital Teaching Symposium. So the next session is in two weeks. Find. Do come along again because we've got a super presentation from Fiona Stewart about TDM or course catalogue and we do still have a gap on the 30th of October, a day before Halloween. So if anyone's got any topic they would be happy to share or equally if there's a topic you'd be interested in seeing here but you don't wouldn't be willing to. Present yourself. Equally do let us know and we can always have an ask around in the community and see if there's anyone who would consider presenting. So do get in touch. So hopefully see you all again in two weeks time and thank you once again to Kelly. Brilliant. Thank you very much. See you next time.

Scumaci, Mary Beth 58:19 Hi, Kelly. Can you hear me? Yeah, I can't turn a camera on. Apparently there's a whole district area. Why Buffalo Public Schools? There's a whole thing around us going on here with teams. So I'm so sorry. I will be watching your playback.

Hambel, Kelly E 58:21 Yes.

Chris Boon (LLE - Staff) 58:22 Yay. Welcome back, Maribeth.

Hambel, Kelly E 58:32 Oh no.

Scumaci, Mary Beth 58:35 What I saw was fantastic and I can't watch wait for the the recording and I can't thank you enough for jumping in and volunteering to present for us today from. I'm just checking out the chat and it looks like everybody had a really good time. So of all the days, right of all the days when when these things happen. So it was the strangest thing. You'd freeze and then I get back.

Hambel, Kelly E 58:44 Of course.

Scumaci, Mary Beth 58:54 Bounced and then I couldn't see the chat and then I couldn't use the mic and then I couldn't use the camera. So I'm not sure what's happening, but we're we're we're feeling it across the campus here. So, but thank you so much for sharing your time.

Hambel, Kelly E 58:59 Oh no. I hope I didn't manifest that because I know I was stressing. I'm like if my computer doesn't work, I'm don't know what I'm gonna do.

Scumaci, Mary Beth 59:09 Nope, Nope. I don't think you manifested that at all. It's just technology not functioning at a a time you'd like it to, right? And it happens. It happens.

Hambel, Kelly E 59:20 Yeah, it knows. It's like this would be inconvenient. Let me not work right now.

Chris Boon (LLE - Staff) 59:20 OK.

Scumaci, Mary Beth 59:24 That's right. That's right.

Chris Boon (LLE - Staff) 59:27 But the the joy of having a team here. So yeah, we're we're all fine. Mary Beth, appreciate you letting us know instead of getting to the end of thinking, where's she gone? Yeah.

Hambel, Kelly E 59:33 Yeah.

Scumaci, Mary Beth 59:35 Where'd you go? I didn't ghost you. Not on purpose anyway. All right, everybody. Thanks for joining us today and have a wonderful week.

Hambel, Kelly E 59:39 Not on purpose.

Chris Boon (LLE - Staff) 59:40 No, exactly. Yeah. Yeah, wonderful. And Kelly, if you just ping over anything, you're happy to to share and we'll make sure it gets uploaded to the community site. No, no rush, but yeah, we can upload it and there's there's clearly lots of people who would be would thoroughly appreciate going through things in detail later.

Hambel, Kelly E 59:47 Yes. Yes. Yes, I will for sure. Thank you.

Chris Boon (LLE - Staff) 1:00:03 OK.

Hambel, Kelly E 1:00:05 Have a good one, everyone. Bye.

Scumaci, Mary Beth 1:00:06 Bye, bye.

Chris Boon (LLE - Staff) 1:00:07 Thanks then. See you later. Bye, bye.

Joni Iglinski 1:00:08 Thanks again. See you now.

Scumaci, Mary Beth 1:00:10 And and I'm curious how this is gonna work, cause Claire said she couldn't be back to turn us off right away, but it should.

Chris Boon (LLE - Staff) 1:00:15 I know that's right. I wonder. I think she said she was going to keep half an eye on things, but I wonder if it will just need to be top and tailed before it's uploaded.

Scumaci, Mary Beth 1:00:21 OK. I think so. So it might take a take a little bit longer, but she must have it open somewhere so it continues with the recording until she can get back because I can see it's still recording.

Chris Boon (LLE - Staff) 1:00:30 Yeah. Yeah, better to capture everything than than miss a bit. So I think that yeah, fantastic. So it might not be the time for a a lengthy debrief, just in case it does can't get top and tail.

Scumaci, Mary Beth 1:00:37 Absolutely.

Scumaci, Mary Beth 1:00:44 I agree. All right, everybody, have a good one. Bye, bye.

Hambel, Kelly E 1:00:45 Yeah.

Scumaci, Mary Beth 1:00:49 OK.

Chris Boon (LLE - Staff) 1:00:49 Great to see you. Bye bye. See you next time. Bye bye.

Carol Chatten 1:00:51 Thanks everyone. Bye.

Carol Chatten stopped transcription