AI Framework Group

Vision

The AI Framework group is focused on how to support other product groups at GitLab with the AI Abstraction Layer, and GitLab AI feature development.

👌 Team OKRs

If you’re interested in the team’s Objectives and Key Results (OKRs), you can find them on GitLab.

🚀 Team Members

The following team members are permanent members of the AI Framework Group:

Name	Role
Martin Wortschack	Engineering Manager, Foundations:Import and Integrate
Alexander Chueshev	Staff Machine Learning Engineer, AI-powered:AI Framework
Alejandro Rodríguez	Senior Backend Engineer, AI Powered:AI Framework
Fabrizio J. Piva	Machine Learning Engineer, AI-powered:AI Framework
Hongtao Yang	Senior Machine Learning Engineer, AI-powered:AI Framework
Kalyani Gadgil (KG)	Backend Engineer, AI Powered:AI Framework
Mark Lapierre	Machine Learning Engineer, AI-powered:AI Framework
Nathan Weinshenker	Backend Engineer, AI Powered:AI Framework
Tan Le	Staff Machine Learning Engineer, AI-powered:AI Framework

Stable Counterparts

The following members of other functional teams are our stable counterparts:

Shared Responsibilities

☕ Team Responsibilities

Team responsibilities include

• Facilitating the integration of AI capabilities throughout GitLab by leveraging the AI Abstraction Layer and AI Gateway.
• Guaranteeing the presence of comprehensive global observability, monitoring, documentation, and enhancements in latency.
• Providing essential support for our AI chat system framework.
• Incorporating support for new AI providers when required.
• Assisting with the production support and ensuring the readiness of the AI Gateway.
• LLM inference support, including prompt enginering, response evaluation, fine tuning, evaluation, and more.

☎️ How to reach us

Depending on the context here are the most appropriate ways to reach out to the IDE Group:

• Slack Channel: #g_ai_framework
• Slack Groups: @ai-framework (entire team) and @ai-framework-engs (just engineers)

📦 Team Processes

📆 Regular team meetings

❗️Important: For every meeting, the AI Framework team’s meeting document should be used, and filled with the meeting notes, as well as references to any other sync meeting agendas/notes/recordings which have recently occurred. This will make it easier for people to find any meeting notes.

Team Meetings

1. Weekly Refinement Assignment Meeting
   • When: Every Monday, alternating between 14:00 AM GMT+1 and 19:00 PM GMT+1
   • What: This meeting replaces the previous Work Assignment Meeting and focuses on refining issues. The Engineering Manager and Product Manager ensure all issues are properly refined.

Shared calendars

• AI Framework PTO (Calendar ID: c_eca9440729dba2cbd88b3117fa70839836fb5811cb072132b94c52f912a31bf5@group.calendar.google.com)
• AI-Powered Stage Calendar (Calendar ID: c_n5pdr2i2i5bjhs8aopahcjtn84@group.calendar.google.com)

AI Framework team members should sync your PTO events with AI Framework PTO calendar.

🖖 Weekly EM Updates

Each week the team EM provides a Weekly Status update issue which aims to capture the most important items for the team to be aware of. These can be found here.

📚 AI Framework Board Outline

Our workflow process involves a weekly board walk where we review issues with the Deliverable label. Here’s how we organize our board:

How do people know what to work on?

An issue is ready to be taken by an assignee when it has all of the following:

• Deliverable label
• The current Milestone
• Either workflow::ready for development or workflow::refinement label

Board Lists

1. Open 📝: This list includes all identified issues. An engineering manager will be assigned if either the Milestone or the “workflow::ready for development” label is missing.
2. workflow::design ✏️: During this phase, issues undergo design refinement. After design considerations are integrated, the “ready for development” label is applied.
3. workflow::refinement 🧪: Issues in this stage are undergoing engineering refinement to ensure the proposed solution meets all requirements. Once refined, the “ready for development” label is applied.
4. workflow::ready for development 🎯: Issues that are prioritized and assigned to a specific milestone are moved to this list, and the “ready for development” label is applied.
5. workflow::in dev 👩‍💻: When a developer begins work on an issue, they should move it to this list and apply the “in dev” label.
6. workflow::in review 👀: After development is complete, the issue moves to this list, and the “in review” label is applied.
7. workflow::verification ✅: Following a successful code and UX review, the issue should be moved to this list and the “verification” label should be applied.
8. workflow::complete 🎉: Once the issue is verified and confirmed to be working properly, it should be moved to this list, the “complete” label should be applied, and the issue should be closed.

🔄 Processes

✍️ Backlog Refinement

Every week the engineering team completes a backlog refinement process to review upcoming issues. The goal of this effort is for all issues to have a weight so we can more accurately plan each milestone using the estimated capacity for the team and the estimated issue weights.

In addition to this backlog refinement process, engineers on the team can add weights to any issues that are straight-forward and do not need backlog refinement.

This process happens in three steps.

Step 1: Identifying Issues for Refinement

The engineering manager picks issues to refine each week. We aim for 5 issues in total. If you think an issue is a good candidate, just mention it in the issue itself

We try to keep refinements themed to avoid too much context switching. Good places to look for candidates:

• Infradev Issues
• Issues scheduled for the next milestone without weight
• Security Issues
• Missed-SLO
• Approaching-SLO
• Bugs
  • Rails
  • AIGW/DWS
  • CEF
• Issues without weight
  • Rails
  • AIGW/DWS
  • CEF

Once selected, the engineering manager applies the ai-framework::ready for next refinement label and uses the Refinement Bot to generate a refinement issue with all the candidates bundled together.

Step 2: Refining Issues

Over the week, each engineer on the team will look at the list of issues selected for backlog refinement. Current backlog refinement issues.

For each issue, each team member will review the issues and provide the following information:

• Estimated weight.
• How to break down the issue into different issues or merge requests.

Some considerations:

• Keep the conversation on the original issues.
• During this process, the issue description and labels should be updated as more information is gathered.
• For efficiency, engineers can also skip the refinement of some issues depending on the feedback that we already have, provided the issue still meets our definition of ready.
• Where the fix is clear and easy, we can assign the issue to ourselves, give it a weight of 1 and push the fix.
• If an issue affects multiple importers, consider an epic, multiple linked issues or an MR for each importer.

An issue is more likely to be scheduled for development if it meets our definition of ready.

📝 Issue Guidelines

These guidelines apply to all issues we use for planning and scheduling work within our group. Our Engineers can define specific implementation issues when needed, but the overall goal for our issues are as follows:

• Provide a meaningful title that describes a deliverable result.
  • ✅ Add the new GitLab Duo chat package as a Vue2 extension
  • ✅ Chat: move away from using OpenAI embeddings
  • ❌ Make Chat better
• Provide a meaningful description that clearly explains the goal of the issue, and provide some technical details if necessary.
• Should there be critical implementation steps or other useful ways to create small tasks as part of the issue, please use a checklist as part of the issue descriptions.
• The issue should have the Deliverable label, milestone and workflow label assigned.
• Design and frontend engineering use one issue. The same issues moves from workflow::design to workflow::refinement to workflow::ready for development. This ensures that there is a single source of truth for customer-facing issues. If a design issue is too large to be implemented, it may be promoted to an epic.

It’s okay to create specific engineering-driven implementation issues for more complex features. These would be called Child Issues and they should always link back to their parent. If one issue would spawn many child issues, consider creating an Epic.

🏋 Weighting and Estimation Process

Weight Guidelines

Issues are weighted using the Fibonacci sequence (0, 1, 2, 3, 5, 8, 13+):

• Weight 0: Smallest issues (typos, minor formatting, simple code changes without tests)
• Weight 1: Simple issues with minimal uncertainty (good for new contributors)
• Weight 2: Straightforward issues requiring multiple code/test updates
• Weight 3: Larger issues with some complexity but manageable scope
• Weight 5: Should typically be broken down; acceptable for large manual updates with low risk
• Weight 8/13+: Placeholder weights indicating need for breakdown; too large or uncertain for immediate implementation

Weight Update Process

Every issue assigned to the upcoming milestone needs to be weighed before applying the Deliverable label by Engineering Manager. Engineering Manager needs to check whether weight is assigned and, in case of the weight being equal or above 5, works on breaking issues down into smaller ones.

Engineering manager and Product Manager are responsible for asking to weight issues assigned for the upcoming milestone during weekly team meetings. They should ask engineers to read issue descriptions before the meeting so they are ready to weight them and ask questions if needed. They can split this process between more than one meeting.

📝 AI Feature Evaluations Guidelines - Evaluate like I am 5

See the Evaluate like I am 5 Project and read the docs here.

👏 Communication

The AI Framework Team communicates based on the following guidelines:

1. Always prefer async communication over sync meetings.
2. Don’t shy away from arranging a sync call when async is proving inefficient, however always record it to share with team members.
3. By default communicate in the open.
4. All work-related communication in Slack happens in the #g_ai_framework channel.

⏲ Time Off

Team members should add any planned time off in the “Workday” slack app, so that the Engineering Manager can use the proper number of days off during capacity planning.

🤙 Ad-hoc sync calls

We operate using async communication by default. There are times when a sync discussion can be beneficial and we encourage team members to schedule sync calls with the required team members as needed.

🔗 Other Useful Links

🌍 AIGW Region Deployments

• 🚀 Staging
• 🌐 Production

📝 Dashboards (internal only)

• Requests per provider
• Error budgets
• AI Gateway SLIs
• GCP quota limits
• LLM Sidekiq completions
• Duo Chat Question Categorization
• Security Issues
• Reliability Issues
• Sentry via CompletionWorker
• Sentry via Feature Category
• Monthly Retros
• Chat QA Evaluation
• Chat REST API Error Ratio
• ITPM per model
• Internal handbook page

📹 GitLab Unfiltered Playlist

The AI Framework Group collates all video recordings related to the group and its team members in a playlist in the GitLab Unfiltered YouTube channel.

View page source - Edit this page - please contribute.