Exploring AI-Enhanced Incident Management and Reporting for Firefighters

Role

Product Designer, Project Lead

My Responsibilities

01. Project Lead

Defining the project roadmap, Expanding the team, and facilitated collaboration with subject matter experts and data science researchers.

02. User Research

Gathered user insights through firefighter interviews and perform desk research to identify report generation challenges.

03. Design & Prototyping

Develop the of proof-of-concept, design the low-fidelity, and high-fidelity prototypes, iterate and refine the solution.

04. Independent Research

Identify and evaluate potential challenges associated with applying AI and NLP to firefighter communication data.

Timeline

Research (Phase 1) : Jan 23 - Apr 23
Ideation (Phase 1) : Apr 23 - Jun 23
Research (Phase 2) : Jun 23 - Oct 23
Ideation (Phase 2) : Oct 23 - Feb 24
POC and User testing (Phase 3) : Feb 24 - Present

Tools

Team

(2) Data Science Researchers
(1) UX designer

Business need

Resource optimization

Incident report generation for firefighters diverts valuable time and resources that could be better utilized for other critical tasks, such as emergency response and mental health recovery.

Accuracy & consistency

Inaccuracies can lead to legal consequences and reputational risks. Firefighters require a system that ensures consistent and precise reporting to mitigate these risks.

Impact

Project Scope

50%

User interviews with 8 firefighters revealed this could accelerate their incident report generation by at least

Individual Impact

+ team expansion

Led to addition of the data science team, to further research and develop the concept MVP

Overview

Problem

Writing incident reports for firefighters is not only time-consuming but also error-prone. It considerably increases their work stress, which has an adverse effect on their psychological well-being and job performance.

Solution

I conceptualized and designed an advanced incident reporting system for the firefighters. It leverages deep learning and Natural Language Processing (NLP) to analyze raw conversation data, automatically extracting key details and summarizing the incident. This generates a high-quality draft report, saving firefighters time and ensuring consistent reporting. After reviewing and making any necessary edits, firefighters can submit the final report with ease.

Teleport to the solution

Ready for the details? Let’s take a closer look

RedLab.ai

IU RedLab (formerly Crisis Technologies Innovation Lab) leads the charge in developing and deploying cutting-edge technologies that empower emergency responders on the front lines to better manage crisis and disaster situations.

Research (Phase 1)

Understanding the problem

This resource-intensive data entry task diverts valuable time and energy that can be better utilized for other critical tasks. Furthermore, the potential legal consequences and reputational risks associated with inaccurate incident reports underscore the urgent need to address this issue.

problem illustration

Research Insights

Ideation (Phase 1)

Product Ideation

Here is a quick walkthrough of my proposed concept

(2 mins watch)

FAQ

What prompted my exploration into AI/ML solutions?

Primary research finding

Primary research revealed that in the current operational flow, incident radio conversations are recorded but remain largely unused, representing an underexplored data resource. This prompted me to ask: How can we effectively leverage this data? Subsequently, I began exploring opportunities within the AI/ML domain.

Literature review

Similar research and implementations exist across various domains: healthcare (clinical data summarization), supply chain (inventory and warehouse report generation), education (lecture note generation), business (GPT implementations) etc.

While the concept may appear straightforward, we made some initial assumptions while ideating the concept. These assumptions necessitated further exploration into technical feasibility to assess the concept's true potential.

One of the major assumption was:

Assumption :

Based on successful implementations in other domains, we hypothesize that this concept could be transferable to the emergency response domain.

Research (Phase 2)

Technical Research

To validate the core assumptions of our concept, I needed a deeper understanding of core AI principles, generative AI, and Natural Language Processing (NLP). Through interviews and collaboration with subject matter experts (data scientists and researchers), I obtained valuable feedback on the proposed concept. This comprehensive research allowed for the identification of critical technical prerequisites to bring the vision to life.

Data preparation

Real data from incidents is required to train and fine-tune the model for generating the required output

Large Language Model

A trained LLM model that can perform the Abstract Dialogue Summarization and data extraction using the input (real time incident data)

Hardware and Infrastructure

Servers, powerful computers that can handle substantial memory demands, stable network.

While previous research and implementations in other domains, have explored similar report generation concepts utilizing abstract data summarization techniques, our domain presented distinct challenges. To address these, I collaborated with the data science team to perform some pilot testing with existing SOTA base LLMs without pretraining to thoroughly analysis and identify and the current constraints and limitations.

Hypothesis

State-of-the-art LLM models trained for abstract dialogue summarization tasks may struggle with the specific challenges of incident data (from radio conversations), leading to reduced model efficiency and inaccurate summary generation (hallucination).

Additional Challenges in Emergency Response Domain

Unique Nature of First Responder Conversations​

Extensive Use of Code Words​

Increased Background Noise and Interference​

Estimating incident duration is complex

Requires substantial and continuously accessible data storage infrastructure

Now lets take a deeper dive in order to understand some of the additional domain challenges:

Pilot Testing

Therefore, the data science team performed some pilot testing to test the hypothesis. They used two SOTA text summarization models: Open AI's GPT 2 and SpaCy.

Initially the models were fed list of commonly used incident terminologies and firefighters codes to build some basic understanding (pretraining). Following that we asked the model to generate an incident summary for a transcript that we prepared from a incident training video (without data scrubbing or data cleansing).

Result:

Unclean raw data packed with firefighters' jargon hampered model accuracy. Concept implementation requires the model to have a solid understanding of firefighter communication in order to handle potential situational challenges such as network issues and noise affecting incident data.

Their final findings matched the initial hypothesis.

Identified technical constraints and limitations

Zero Dataset

Zero Research

Zero Benchmarks

Added Domain Challenges

Intensive Need for Funds

Zero Dataset

No publicly accessible datasets that include actual incident conversation data (either recordings or transcripts).

Zero Research

No prior research on abstractive dialog summarization for Fire & Emergency service domain

Zero Benchmarks

Due to the absence of prior research, there is No established benchmarks.

Added Domain Challenges

Unique domain challenges makes state-of-the-art (SOTA) models ineffective against incident dialogue summarization

Intensive Need for Funds

Exploring this uncharted domain requires significant financial investment, due to its novelty and complexity.

But hold on, first we must identify our Success Criteria's:

Efficiency

Significantly speeding up the report
generation process. Allocating time for critical stuff and reducing stress.

Testing: Model Algorithm
Tasked to: Data Science Team

Accuracy

Precision is crucial in fire and safety domains as inaccuracies may cause disruption and fatal consequences.

Testing: Model Output
Tasked to: Data Science Team

User Behavior & Adoption

Concept validation and adoption; user trust impacts adoption, but over-reliance risks errors.

Testing: Prototype (wt. Firefighters)
Tasked to: UX Team

Ideation (Phase 2)

Brainstorming the frontend

Proof of Concept (Phase 3)

High fidelity prototype (v3.0)

Through rigorous design sprints with the team, we iteratively refined our initial designs, gathering valuable feedback from both stakeholders and firefighters. Over three iterative rounds, we refined each detail to create the high fidelity prototype.

01  Advanced incident management dashboard

Integrating all-in-one cohesive enterprise solution

Transform firefighting operations with our integrated incident management platform. Personalize your workspace with widgets that deliver real-time analytics, putting crucial information at your fingertips. Further embrace AI integration for limitless potential in enhancing operational efficiency and safety.

02  New incident reporting flow

Simplified workflow and amplified efficiency

Optimized NFIRS user flows, ensuring intuitive navigation and minimal disruption. Your team can effortlessly leverage advanced technology without missing a beat in their critical operations.

03  Firegen: AI integration

AI automation for precise, swift incident documentation

Experience seamless incident reporting with our cutting-edge AI system. Analyzing raw radio data, it automatically extracts essential information and generates a comprehensive incident summary. Firefighters can swiftly validate and submit an auto-filled draft, streamlining the reporting process with accuracy and efficiency

04  Firefighter-centric design

Enhanced experience with easy editing

Designed with firefighters' mental models in mind, the platform simplifies data entry for firefighters by offering a dedicated section for editing all form field values. Select the field, move to the edit response section, make changes, and save (auto).

Assumption :

User might get confused regrading which values might have been auto-filled by the AI and could lead to error.

05  Keyword-based data categorization

Introducing novel keyword categorization

To optimize AI-driven report generation, we've introduced an innovative keyword-based categorization system.

We classify incident transcript data into five key categories:

Identifiers (e.g., "Street; Intersection; Incident"); Extraction Data (e.g., "West Michigan St."); Sectional Keywords (e.g., "Arrived at location"); Metadata (e.g., fire department details); and Filler Data (anything that cannot be classified or situational content).

This structured approach enhances data processing efficiency, ensuring vital information is accurately extracted for streamlined reporting.

Testing wt. stakeholders & team

I presented the prototype (v3.0) designs to the stakeholders and the project team to gather initial feedback before testing with firefighters. The stakeholders raised some insightful questions about the current designs that we will be validating during user testing with firefighters. They also shared some ideas that could improve platform usability.

#1  Source representation

Do firefighters want to see the source?

Would they prefer a link to the actual transcript?

Rationale behind the existing design approach:

provide enough detail to the firefighter to validate the data extracted by the AI without overwhelming them or adding cognitive load

#2  Edit response

Is it easier for firefighters to have a separate section to edit response (validate assumption) ?

Rationale behind the existing design approach:

simpler workflow and prevent possibilities of unintended errors (act as two-factor verification)

#3  Response change history (idea)

What if someone wants to verify who made specific changes (validate use case)?

Rationale behind the existing design approach:

use case was not identified initially

Future: the AI Infinity

This project marks a pivotal moment in leveraging the latest advancements in AI/ML in the fire and safety domain. By addressing critical domain challenges and collaborating closely with fire departments, we aim to pioneer new frontiers. Training AI models like LLM promises enhanced accuracy and customization, setting benchmarks for future advancements.

Here are some potential future directions for AI integration into the Incident Management platform:

01  Diverse report generation

Automatically generate a variety of comprehensive reports tailored to different stakeholders and needs.

02  Anomaly Detection

Quickly detect anomalies in incident data, enabling proactive management and response to unexpected situations.

03  Individual performance analysis:

Utilize AI to analyze and provide insights into the performance of individual firefighters based on incident data and historical trends.

04  Team performance enhancement

Provide actionable insights and recommendations to improve overall operational performance of the team based on AI-driven analysis of incident handling and response strategies.

Next Steps

Ongoing

Next we need to rigorously test the prototype (v3.0) with firefighters. This testing will determine if this approach aligns with their mental models and effectively balances ease of use with the required control.

Long Term Milestones

Engaging with Subject Matter Experts (SMEs) allowed me to identify key long-term milestones that would be crucial for shaping the project.

Learnings

Navigating Leadership

Spearheading the project from start taught me the vital role of effective leadership. Successful project completion hinges on skills such as effective communication, task delegation, informed decision-making, and most importantly, the willingness to take responsibility, including accepting failure when necessary.

Data Driven Approach

One of the key learnings from this project was the importance of having a data-driven approach. A major accomplishment I had in this project was effectively demonstrating the project's potential to clients through my research-backed project proposal and impactful visuals, which actually led to team expansion and additional resource allocation.

Collaborative Synergy

Cultivated essential interpersonal skills through collaboration with diverse professionals (firefighters, researchers, entrepreneurs, and data scientists). Emphasized active listening, empathy, and personal connection to foster collaboration, cultivating positive team dynamics.

Other works

Link to ESUP
Link to crayons

let's connect & create something together

Thanks for reaching out. I will get back to you soon!
Drop a mail:  anamitrajana@outlook.com
Oops! Something went wrong while submitting the form.

#halamadrid