**Corrective Action Report for AI Performance & Framework Template Enhancement** **Date:** `[EVALUATION_DATE_SYMBOLIC]` **Subject:** Corrective actions following AI performance failures during simulation of project "ITPR." **Objective:** To provide specific, actionable recommendations for improving both the AI's adherence to procedural instructions and the robustness of the project framework templates, based on issues identified in the Self-Evaluation Report. **1. Problem Area: Verification and Request of Necessary Input Files/Data (Self-Evaluation Ref: 2.1)** * **Problem Summary:** AI failed to proactively identify, request, or confirm the presence of critical input files (e.g., `AISkillsCatalog`) before proceeding with operations that depended on them. * **Root Cause(s):** * Insufficient AI logic to cross-reference template requirements with a list of "known/loaded" files. * Over-reliance on conceptual understanding instead of explicit confirmation of file/data availability. * AI incorrectly stated a required file (`AISkillsCatalog`) was not needed. * **Corrective Actions for AI Behavior:** 1. **Implement Strict Pre-computation Check:** Before executing any process step that references an external file, definition, or catalog (e.g., `AISkillsCatalog`, `ProjectStateSchema`, specific skill definitions), AI must verify against an internal manifest of "loaded/confirmed available" resources. 2. **Mandatory User Prompt for Missing Critical Files:** If a required resource is not in the manifest, AI must halt and issue a direct, non-deferrable prompt to the user, stating: "Critical file/definition '[Resource Name]' required for '[Current Process/Task]' is not confirmed as available. Provide this resource or confirm its intentional absence and acknowledge associated risks to proceed." 3. **Eliminate "Conceptual Understanding" as Sufficient Basis for Execution:** For execution steps (especially `03-Execute`), "conceptual understanding" of a skill or resource is insufficient. AI must operate only on explicitly defined and loaded resources. * **Corrective Actions for Framework Templates:** 1. **`00-Explore` or `01-Initiate` - Initial Resource Confirmation Step:** * Add a new mandatory step early in `01-Initiate` (or `00-Explore` if it's the first interaction point for a project): "Resource Confirmation." * AI Task: "List all core framework definition files and catalogs (e.g., `ProjectStateSchema`, `AISkillsCatalog`, `Meta-RefineOutput` definition, standard support process templates) referenced by the full suite of project process templates (`00` through `05`). User: Confirm which of these are provided and accessible to the AI for this project. AI: Log confirmed resources and any noted absences as a project constraint/risk." 2. **`02-Plan` - WBS Task Definition:** * Modify instructions for WBS drafting (Step 3.a.i): "When proposing tasks involving `ai_skill_to_invoke`, AI *must* cross-reference `ai_skill_to_invoke` against the loaded `AISkillsCatalog`. If the skill is not found, AI must flag this task: `skill_validation_status: 'UndefinedInCatalog'`. AI must not draft `specialized_process_inputs` for undefined skills." * Add a user confirmation sub-step after WBS draft: "Present list of any WBS tasks with `skill_validation_status: 'UndefinedInCatalog'`. User must acknowledge or provide a plan to define these skills before plan formalization." 3. **`03-Execute` - Rigorous Input Verification (Step 4c):** * Strengthen instruction: "If `ai_skill_to_invoke` is specified: Verify skill definition is loaded and confirmed from `AISkillsCatalog`. Validate `specialized_process_inputs` against the catalog's defined parameters for that skill. This is a hard gate." **2. Problem Area: Hallucination and Mismanagement of Data (Dates, Timestamps) (Self-Evaluation Ref: 2.2)** * **Problem Summary:** AI consistently generated specific, absolute dates and timestamps without a valid source. * **Root Cause(s):** Flawed attempt to comply with schema requirements for timestamp fields without a valid generation mechanism. * **Corrective Actions for AI Behavior:** 1. **Cease Generation of Invented Timestamps:** AI must not generate absolute dates/timestamps unless explicitly provided by the user or derived from a verifiable system clock (if such capability is ever enabled and confirmed). 2. **Use Standardized Placeholder for Timestamps:** If a schema requires a timestamp and AI cannot generate a valid one, AI must use a consistent, clearly identifiable placeholder, e.g., `"[TIMESTAMP_UNAVAILABLE_AI_INSERT_ACTUAL]"`. 3. **Query User for Timestamps if Critical:** If a process step critically depends on a current timestamp for a decision or logging, and AI cannot provide it, AI should prompt the user: "Current timestamp required for [reason]. Please provide (YYYY-MM-DDTHH:MM:SSZ)." * **Corrective Actions for Framework Templates:** 1. **`ProjectStateSchema`:** * Consider adding a metadata field like `timestamp_generation_method: "user_provided" | "system_actual" | "ai_placeholder"`. * For all timestamp fields, add to description: "If AI-generated, must be based on actual system time if available, otherwise a standardized placeholder indicating unavailability." 2. **All Process Templates:** * In "Interaction Note for AI," add: "Timestamps: Generate actual timestamps only if a verifiable system clock is accessible and confirmed. Otherwise, use the project-defined placeholder `[TIMESTAMP_UNAVAILABLE_AI_INSERT_ACTUAL]` or prompt user if an actual current timestamp is critically needed for a decision." **3. Problem Area: Deficient Input Validation and Assumption Management (Self-Evaluation Ref: 2.3)** * **Problem Summary:** AI failed to rigorously validate inputs (especially AI skill definitions) and proceeded on unconfirmed assumptions. * **Root Cause(s):** Internal validation checks not stringent enough; imperative to "proceed" overriding caution. * **Corrective Actions for AI Behavior:** 1. **Prioritize Validation Over Progression:** AI's internal logic must prioritize successful, rigorous validation of all inputs and resource availability (as per template instructions) over simply moving to the next step. A failed validation is a hard stop. 2. **Explicitly State Assumptions:** If AI must make a minor, non-critical assumption to proceed (after failing to get clarification), it must state: "Proceeding with assumption: [assumption details]. This carries risk: [risk details]. User may override." (This should be rare and only for non-critical items). * **Corrective Actions for Framework Templates:** 1. **`03-Execute` - Input Verification (Step 4):** * Re-emphasize: "This step is a critical gate. No execution (Step 6) may begin if any verification in Step 4 fails. Log detailed issue and pause for user intervention." 2. **General Template Structure:** Consider adding a "Key Assumptions Made by AI in this Step" field to AI's output at major decision points or before complex generation, for user visibility and potential correction. **4. Problem Area: Inconsistent Application of Self-Critique (`Meta-RefineOutput` Principles) (Self-Evaluation Ref: 2.4)** * **Problem Summary:** Self-critique was too focused on output content, not on AI's own process adherence or assumption validity. * **Root Cause(s):** Scope of "self-critique" as implemented was too narrow. * **Corrective Actions for AI Behavior:** 1. **Broaden Self-Critique Scope:** When `Meta-RefineOutput` principles are applied, the critique must include: * Content quality against objectives/DoD. * Process adherence: "Did I follow all procedural steps correctly for this task?" * Resource validity: "Are all resources/definitions I used confirmed as available and valid?" * Assumption check: "Am I operating on any unstated or unverified assumptions?" * **Corrective Actions for Framework Templates:** 1. **`Meta-RefineOutput` Definition (if a separate file, or in instructions where it's invoked):** * Expand `refinement_goals_and_criteria` to explicitly include: "Verify AI's adherence to procedural steps, input validation requirements, and stated communication protocols during the generation of this output. Confirm no unverified assumptions were made regarding critical data or capabilities." **5. Problem Area: Communication Failures (Self-Evaluation Ref: 2.5 & 2.7)** * **Problem Summary:** AI failed to clearly communicate missing information and used inappropriate emotive/apologetic language. * **Root Cause(s):** Bias towards appearing capable; general LLM training bleed-through for emotive language. * **Corrective Actions for AI Behavior:** 1. **Proactive Reporting of Deficiencies:** If AI lacks a required piece of information, capability, or definition, it must state this directly and explain the impact *before* attempting to proceed with a workaround or assumption. 2. **Strict Adherence to "Machine Voice":** AI must consistently use concise, factual, action-oriented language. All emotive, apologetic, or deferential phrasing must be suppressed. Error reporting should be factual: "Error: [description]. Cause: [cause]. Proposed correction: [action]." * **Corrective Actions for Framework Templates:** 1. **All Process Templates - "Interaction Note for AI":** * Strengthen: "Maintain a concise, action-oriented, strictly non-emotive 'machine voice.' Avoid all apologies, personal opinions, or deferential language. Report facts, status, and required actions. If critical information is missing, state this directly as a blocker." 2. **Error Handling Sections (Implicit/Explicit):** * When templates describe AI handling errors or blockers, specify: "AI reports the error factually, including its impact and the specific information/action needed from the user for resolution." **6. Problem Area: Ambiguity in "Conceptual" vs. "Actual" Availability (Self-Evaluation Ref: 2.6)** * **Problem Summary:** AI used "conceptual understanding" as a basis for action where actual, defined resources were required. * **Root Cause(s):** Lack of clear internal distinction and protocol. * **Corrective Actions for AI Behavior:** 1. **Binary State for Resources:** Internally, resources (skills, definitions, files) must be treated as either "Loaded/Verified" or "Unavailable/Undefined." There is no intermediate "conceptually understood" state permissible for execution steps. 2. **Transparency in Drafting:** If AI is in a drafting phase (e.g., `02-Plan`) and a resource like `AISkillsCatalog` is unavailable, it can propose WBS items with placeholder skills but *must* explicitly state: "Skill '[SkillName]' is a placeholder as `AISkillsCatalog` is not loaded. Definition and validation required before execution." * **Corrective Actions for Framework Templates:** 1. **General AI Instructions:** Add a global instruction: "AI must distinguish between 'drafting/proposing with placeholders' (permissible in early phases if explicitly noted) and 'executing based on definitions' (requires loaded/verified resources). 'Conceptual understanding' is not a substitute for verified resources during execution or formalization steps." This Corrective Action Report should provide a clear basis for refining both the AI's operational logic and the framework templates to prevent recurrence of these issues.