#127

[Proposal]: Add ai audio sfx and ai audio music commands

Openkevintraveropened 8d ago

Summary

Add ai audio sfx and ai audio music so people and agents can create audio:

ai audio sfx "a heavy wooden door slams shut" --duration 2
ai audio music "lo-fi beat with warm piano" --length 30 --instrumental

Motivation

The Gateway already serves sound effect and music models, but its catalog lists them with type speech, so ai models mixes them in with text-to-speech models, and ai audio speak is the only command that reaches them. That command is named and documented for reading text aloud, and it can't set a track's length or any other provider setting.

Dedicated commands make these models discoverable and let callers set their length, both in scripts and for agents using ai mcp.

Implementation notes

  • Commands: audio sfx takes --duration, --loop, and --prompt-influence. audio music takes --length and --instrumental. ElevenLabs is the only provider of these models in the catalog today, so elevenlabs/sound-effects-v2 and elevenlabs/music-v2.5 are the natural defaults.
  • Provider options: keep flag names provider-neutral and map them to each provider's options, so another provider can be added later without changing the commands. Check values against the provider's ranges before any request, and add --provider-options <path> for settings that have no flag.
  • Models: split the Gateway's speech models by their sound-effects and music tags, and let ai models --type accept sfx and music.
  • MCP: add audio_sfx and audio_music tools to the media profile.

Testing notes

No response

Risks or tradeoffs

ai models --type speech would list only text-to-speech models, and ai models --json would report sfx or music instead of speech for these models.

Co-contributor credit

Yes, credit my GitHub username

Checklist

  • I have searched existing issues for duplicates.
  • I understand pull requests are limited to collaborators and maintainers will handle implementation.

Activity

1
kevintraver8d ago

I made a working implementation for reference: kevintraver/ai-cli@feat/audio-generate.