print(f'Hello Agent Zero folks! \nRead the whole post please!')

Greetings! I am a fresh new member to this site and group. I've been starting to work with A0 for the last week or so. It's a work related project, to ultimately build a local A0 deployment that can act as a Systems Admin essentially. With limits of course! I am also new to the world of AI, yes I know I'm late to the party. But its become clear that AI is here to stay.

I'm looking to collaborate with other A0 engineers that are using local LLM's to host A0 (not paid cloud based API send all your data to a random company and hope for the best for privacy) that are working on similar types of "system admin" level use of A0 and creating guardrails. I mean I don't care if you use cloud models, I don't think that impacts the guardrails as much, maybe it does I dunno. Again I'm 2 weeks into this whole world of AI. It's information overload. But building out rules is where I would love to collab with people. Skills too. More about me below.

Prior to about 2 weeks ago I knew nothing about AI, I used chatGPT once or twice for fun, then I used my employers enterprise Co-Pilot subscription we have to help me as basically a faster/better google search in my daily duties. That's it. I didn't know how anything worked.

Sad I know especially not being new to tech which I've been nerding out on my whole life really, well since the internet got big anyways (I am part of the last generation to grow up as a kid without internet.) It wasn't until my mid teen's until the 90's boom happened and I was hooked. 20+ year veteran of being a Senior systems admin/architect worked for some of the biggest tech companies in silicon valley and some government work too.

My employer has given me an NVIDIA DGX Spark 10 to use at home to build out a local deployment of A0, backed 100% by local LLM's. No internet access needed!

I've been diving into AI to learn the fundaments and to say its overwhelming is an understatement. Having an engineering degree, I have the brain for complex tasks and puzzles so AI is just that to me. I first had to get a handle on the DGX Spark. I'm ok at Linux but no guru. I'm running the spark headless. As that's how it'll be used at work. I'm using A0 on my local laptop for testing via Docker desktop.

So far after playing with a ton of LLM's I found a mix that so far is working well for me on the Spark and A0. The Spark is running a few inference engines to host different models for A0. Honestly I could combine them and probably will, but I'm still learning and who knows I may end up somewhere else in terms of inference engines and LLMs nothing set it in stone.

I'm using vLLM a forked version of NVIDIA's official one that has some fixes for some of the issues NVIDIA hasn't fixed yet (basically the Spark is so new that their own software stack hasn't caught up to the hardware, they have their own flavor of vLLM which well is not there yet) for my main inference engine that runs my main chat model for A0.

I'm also using the command line version only of LM Studio on the spark to host my utility model for A0. And localai to host a smaller embedding model.

I'm getting pretty good speeds now, fast responses, 40-50 tokens per second and a 70% key value cache hit rate, which is pretty darn good. For the Spark architecture, I'm running big models.

I am now working to start the long and painful process of setting up guardrails in A0. The first one I'm taking on, internet access --meaning controlling when A0 can reach out to the web vs not. We are doing zero trust, least privilege type work, or I should say, that is what we want A0 to behave as. So I've started to configure it to be set to not call out to the internet unless it is given permission to due so. I've gotten that working. I also had an issue with my utility model running out of context length despite a large size on the backend, that has been solved in the configuration files of A0. I've had to modify several of the .MD files on the system to add the gates, and even edit the agent.py file for some more utility caching control with some extra lines of python. (oh yeah I know a little python from learning micro-python when messing with microcontrollers as a hobby)

I'm trying to get it right now to display a banner whenever a request calls the web tool after being given permission. Meaning it should let the user know its using the web. I can see in the docker logs (of A0) that it see's the rule but doesn't honor it. Co-Pilot is helping me out, and its saying that the way v1.6 works its never going to work right and the best it can do is post a msg saying the web was used. It's saying this doable in v1.7 but there is no 1.7!! (Hence I never trust AI) But so far I'm happy I've gotten it to default to no web usage, and ask for access, if its own knowledge can satisfy the question correctly. It's also tracking the context of the question so if given access to the web to solve say a Microsoft problem, and then I ask it what the current price of a stock is, it knows the scope changed and prompts for web access after giving me some generic info about the company it doesn't assume it still has permission to use the web.

Anyone else doing this kind of stuff? I'm going to have to setup up all sorts of rules or gates if you will not only for web, but for system access and not letting it just go bananas on a server. So to say I have a lot of configuration to do is an understatement!

I'm looking to work with like minded folks who are working on skills, and guardrails and how you are doing it. I've been doing 99% of it in MD files by adding "rules" it must abide by. I know it can ingest skills to, which is something I would love to do but I dont know where to find any reliable, trustworthy skills or knowledge markdown (.md) files. I do have a way of downloading a website and turning into and .md file pretty easily with python, which I can use to feed it knowledge but I don't know about skills. I'm honestly thinking about buy cheap e-books that cover entire subjects (current) say Cisco networking and converting them into .md files and feeding them as knowledge. I don't know I'm spitballing ideas around. This is why I want to see if anyone out there is doing this kind of stuff (surely not everyone in here is just letting A0 run wild with no guardrails, that would be scary.) I'm not looking to integrate social media or things like that, this isn't a project to make money its sole purpose should be:

act and think like a very senior systems admin
network admin
security admin
design cloud and on prem solutions
virtual platforms (VMware, xcp-ng, proxmox)
storage admin
SDN design implement
monitoring and alerting
create documentation
automation of all sorts of tasks
can use orchestration tools like ansible and terraform for IaaS
light coding for scripts

All tightly controlled so it can't just go and do things without permission. Unless we set it up to do so like for alerting and maybe even taking actions on alerts it'll need permission to do that on its own after a lot of testing of course.

Any and all resources out there that might help please share! Also I'm really hoping to have a small tight nit group of A0 engineers to collab with who aren't going to flake out or are just 'playing for fun' with A0, this is eventually going to be a production system at my job so I'm taking seriously. I'm at a deficit in that I'm new to A0, All things AI, so looking for people who are OK with stupid questions. I of course will do my best to help you and collab on your work too. No sensitive work data will be passed around, its about configuration and tuning of A0. I have to do this so I'm locked in. I can't bail out lol. So yeah. I know this is a super long post. But I wanted to give you my history and where I'm headed and hopefully headed there with one or more I hope, people to become a group that can really tweak A0 to our liking and needs.

For now I'll be lurking. Tomorrow I'm going to try and tackle the internet in use banner issue. See what progress I make. If any. Expect questions to be posted as I continue on! For what it's worth I have my work's enterprise Co-Pilot AI available to me to help. It's good but not perfect and can lead you down the wrong road if you aren't staying sharp (as is of any AI). It beats googling all day long sometimes. I can't give you access to it of course, but if we have a related issue and are stuck maybe we ask the robots how to help us build robots! lol. Alright thanks for reading to the end. I hope to make some new I.T friends aligned to my mission.

Cheers mates!

1 comment