This is a planning + deploy bug for Universal Search recommendations. Things to conclude: - configuration for apps (env vars) - deployment infrastructure - hostname for stage - CDN requirements - release process

Infrastructure: - CDNs: TBD, we want HTTP2, but for stage the latency is ok - Web tier (ELB -> ASG -> EC2 - Docker - Worker tier, ASG, celery worker - Backing services: memcache and redis. We will use ElastiCache for these For stage we will go forgo the CDN until we can figure out the best solution. The desire here is to reduce latency with closer POPs and high cache hit rates. Release Process: - same as Test Pilot: - tag a release - file a bug - release deployed to stage - release is tested - release is promoted to prod Deploy Infrastructure (stage): - ELB + autoscaling group + EC2 web server - autoscaling group + EC2 worker - ElastiCache Memcache instance - ElastiCache redis instance - Prod will include a CDN to cache responses Hostnames: - universalsearch.stage.mozaws.net - prod: TBD Configuration: - https://github.com/mozilla/universal-search-recommendation/blob/master/recommendation/conf.py#L16-L17 - only two are required: - CELERY_BROKER_URL: e.g: "redis://($HOST):($PORT)/0" - MEMCACHED_HOST: e.g. "($HOST):($PORT)" CDN Requirements: - will review with OPs team to determine which ones are supported :chuck please review the above plan and let me know if I missed anything or if anything is not accurate.

- More configuration is required. I'll securely send you the values for these: https://github.com/mozilla/universal-search-recommendation/blob/master/.env.dist - Prod hostname suggestion: universalsearch.testpilot.firefox.com - The worker is intended to be an identical deployment of the web container with a different entry: https://github.com/mozilla/universal-search-recommendation/blob/master/conf/worker.sh - App currently assume a single memcached server. If you deem it appropriate to cluster, we can allow for that. - I thought I heard rumors of deploying from branches, rather than tags? Happy to do either. Otherwise looks great!

- memcache cluster support would be good for better reliability and load balancing - there is an autodiscovery API for memcache nodes (http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/AutoDiscovery.html) - should implement this, would probably need a third party library - Still figuring out deploy from branches details. Tags for now. That should be enough for me to get started.

Just wrapped up some Docker/Dockerflow tweaks [1]. The docker image is ready to be deployed to stage. [1] https://github.com/mozilla/universal-search-recommendation/pull/70

Deployed. https://universalsearch.stage.mozaws.net

I'm seeing these errors on the worker box: Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: [2016-03-25 04:51:51,201: ERROR/MainProcess] Task main.recommend[e44cf546-e034-495c-8b88-95d4e0621263] raised unexpected: KeyError(1,) Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: Traceback (most recent call last): Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 240, in trace_task Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: R = retval = fun(*args, **kwargs) Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: File "/app/recommendation/factory.py", line 41, in __call__ Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: return TaskBase.__call__(self, *args, **kwargs) Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 438, in __protected_call__ Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: return self.run(*args, **kwargs) Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: File "/app/recommendation/tasks/task_recommend.py", line 11, in recommend Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: recommendation = SearchRecommendation(q).do_search(q) Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: File "/app/recommendation/search/recommendation.py", line 83, in do_search Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: self.suggestions = self.get_suggestions(query) Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: File "/app/recommendation/search/recommendation.py", line 48, in get_suggestions Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: return self.get_suggestion_engine()(query).search(query) Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: File "/app/recommendation/memorize.py", line 106, in wrapper Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: value = fn(*args, **kwargs) Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: File "/app/recommendation/search/suggest/base.py", line 30, in search Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: return self.sanitize(self.fetch(query)) Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: File "/app/recommendation/search/suggest/bing.py", line 27, in sanitize Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: return results[1] Mar 25 04:51:51 ip-172-31-1-151 docker[2923]: KeyError: 1 Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: [2016-03-25 04:52:01,129: ERROR/MainProcess] Task main.recommend[91b894a8-e178-4dd8-b6cc-fafb4f3ff63c] raised unexpected: KeyError(1,) Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: Traceback (most recent call last): Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 240, in trace_task Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: R = retval = fun(*args, **kwargs) Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: File "/app/recommendation/factory.py", line 41, in __call__ Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: return TaskBase.__call__(self, *args, **kwargs) Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 438, in __protected_call__ Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: return self.run(*args, **kwargs) Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: File "/app/recommendation/tasks/task_recommend.py", line 11, in recommend Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: recommendation = SearchRecommendation(q).do_search(q) Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: File "/app/recommendation/search/recommendation.py", line 83, in do_search Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: self.suggestions = self.get_suggestions(query) Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: File "/app/recommendation/search/recommendation.py", line 48, in get_suggestions Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: return self.get_suggestion_engine()(query).search(query) Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: File "/app/recommendation/memorize.py", line 106, in wrapper Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: value = fn(*args, **kwargs) Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: File "/app/recommendation/search/suggest/base.py", line 30, in search Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: return self.sanitize(self.fetch(query)) Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: File "/app/recommendation/search/suggest/bing.py", line 27, in sanitize Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: return results[1] Mar 25 04:52:01 ip-172-31-1-151 docker[2923]: KeyError: 1

Those look like regular old application errors, no problem on your end.