Closed Bug 1085739 Opened 10 years ago Closed 10 years ago

[tracking] Improve response time of homepage feed API endpoint

Categories

(Marketplace Graveyard :: API, defect)

Product:
Marketplace Graveyard
Component:
API
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cvan, Unassigned)

References

(
URL
)

Details

(Keywords: perf) 

Eyeballing the traffic from my Runscope timings today, the Marketplace production feed API endpoint from the CDN took 2.352 seconds @ the 99th percentile. (The requests are being made from boxes in US East and Brazil, with averaged timings.)

https://marketplace.cdn.mozilla.net/api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0


CHRIS VAN
Firefox Marketplace

URL                              Status  Time
---                              ------  ----

marketplace.firefox.com          200     255 ms             
GET /robots.txt      299 bytes  3s ago

marketplace.cdn.mozilla.net      200     581 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 55s ago

marketplace.cdn.mozilla.net      200     581 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 31s ago

marketplace.cdn.mozilla.net      200     1641 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 32s ago

marketplace.firefox.com      200         259 ms             
GET /robots.txt      299 bytes  29s ago

marketplace.firefox.com      200         262 ms             
GET /robots.txt      299 bytes  33s ago

marketplace.cdn.mozilla.net      200     601 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 2m ago

marketplace.cdn.mozilla.net      200     1906 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 2m ago

marketplace.cdn.mozilla.net      200     740 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 7m ago

marketplace.cdn.mozilla.net      200     1580 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 7m ago

marketplace.cdn.mozilla.net      200     2158 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 12m ago

marketplace.cdn.mozilla.net      200     1600 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 12m ago

marketplace.cdn.mozilla.net      200     2026 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 17m ago

marketplace.cdn.mozilla.net      200     1746 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 17m ago

marketplace.cdn.mozilla.net      200     2476 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 22m ago

marketplace.cdn.mozilla.net      200     1708 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 22m ago

marketplace.cdn.mozilla.net      200     2386 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 27m ago

marketplace.cdn.mozilla.net      200     1622 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 27m ago

marketplace.cdn.mozilla.net      200     582 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 32m ago

marketplace.cdn.mozilla.net      200     1890 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 32m ago

marketplace.cdn.mozilla.net      200     1549 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 37m ago

marketplace.cdn.mozilla.net      200     2644 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 37m ago

marketplace.cdn.mozilla.net      200     2367 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 42m ago

marketplace.cdn.mozilla.net      200     1692 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 42m ago

marketplace.cdn.mozilla.net      200     1940 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 47m ago

marketplace.cdn.mozilla.net      200     2391 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 47m ago

marketplace.cdn.mozilla.net      200     2234 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 52m ago

marketplace.cdn.mozilla.net      200     1779 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 52m ago

marketplace.cdn.mozilla.net      200     2498 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 57m ago

marketplace.cdn.mozilla.net      200     1605 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 57m ago

marketplace.cdn.mozilla.net      200     733 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     1637 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     1660 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2390 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2304 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     1842 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     1777 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     1583 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     1945 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2206 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     1592 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2647 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2120 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2500 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2159 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     1983 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2378 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2165 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     205 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2078 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     569 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     1788 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2481 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     1714 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.1 kB 1h ago

marketplace.cdn.mozilla.net      200     2309 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2157 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     1948 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     1929 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2393 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     1974 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2423 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     1787 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     1842 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2117 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     1627 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2343 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2494 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2111 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2317 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     1800 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2087 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     1849 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2074 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2517 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     92 ms              
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2598 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2326 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     1695 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    2h ago

marketplace.cdn.mozilla.net      200     2326 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     1897 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     582 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     1950 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     2380 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     2000 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     572 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     1576 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     1849 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     2164 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     595 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     1470 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     581 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     1713 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     1959 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     2302 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     2186 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     1728 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     194 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     2326 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     155 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     2364 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    3h ago

marketplace.cdn.mozilla.net      200     2573 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     1581 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     1849 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2451 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     591 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     1768 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2399 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     1887 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     1789 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2286 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     578 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2066 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2095 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     1571 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2276 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2030 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2496 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     1563 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2872 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2253 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2300 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     1511 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2243 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     1703 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    4h ago

marketplace.cdn.mozilla.net      200     2413 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago

marketplace.cdn.mozilla.net      200     1953 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago

marketplace.cdn.mozilla.net      200     582 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago

marketplace.cdn.mozilla.net      200     1979 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago

marketplace.cdn.mozilla.net      200     570 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago

marketplace.cdn.mozilla.net      200     1438 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago

marketplace.cdn.mozilla.net      200     582 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago

marketplace.cdn.mozilla.net      200     1750 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago

marketplace.cdn.mozilla.net      200     1842 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago

marketplace.cdn.mozilla.net      200     130 ms             
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago

marketplace.cdn.mozilla.net      200     2095 ms                
GET /api/v2/feed/get/?lang=en-US&limit=10&region=us&cache=1&vary=0       4.09 kB    5h ago
Are these being intercepted by the CDN, or passing through to the application server? I'm not sure what options we have to improve response time in requests that never hit our servers.
Most of the requests are cache misses, it turns out. I guess we really don't have that many US users :(

The requests that return cached responses have 'X-Cache: HIT' as response headers in the Runscope request history. (It's really neat that I can look that up.)

For the requests that were cache hits (all the ones in the ~580 ms range), I'm not sure what we could do. Let's talk to IT and get creative - and make sure that in fact the Akamai is telling the truth about its responses being cached. (SPDY is definitely worth investigating. Perhaps other CDNs in particular regions?)

I'll be sure to measure prod (Zamboni) directly.
I would love to work on a CDN that supports prefetching; our needs here are only going to get more complex and our cacheability will go down, while our queries are already pretty tight. I wonder if these numbers are slow enough to get leverage to do so. Any thoughts, Wil?

We could also probably stand to increase our TTL. I think it's at 180s right now.
Agreed about the TTL: The homepage feed API endpoint could stay cached longer, IMHO. 

We do need to keep in mind that any TTL will apply to both the browsers and the CDN though, potentially doubling the time during which users potentially see stale content.
Given the content publication schedule (once a day), there's no reason to not up the TTL. Do we just ask Jason to do that?
No, that's something we control in the django code directly. We just need something a little smarter than what we do currently (set max-age=180s if cache=1 in query parameters).

I'd be in favor of raising the TTL on that endpoint specifically to 3 or even 6 hours. Any concerns from anyone ? Who has final say on something like this ?
(In reply to Mathieu Pillard [:mat] from comment #6)
> I'd be in favor of raising the TTL on that endpoint specifically to 3 or
> even 6 hours. Any concerns from anyone ? Who has final say on something like
> this ?

(1) You do :) Let's do 6 hours.

(2) Bug 957383 is something I've wanted to do so that Fireplace can set a *request header* of `Cache-Control: max-age=21600` for the API responses we want to cache, and then Zamboni takes that and sets that same header in the response, which the CDN then finally respects. (We could then remove the `?cache=` from the query string.)

(3) How can we avoid CDN cache misses? A request that results in a CDN cache miss will always take longer than if the URL was requested directly from Zamboni. I talked to Jeremy about CDN cache prefetching earlier this year, but we decided to not worry about it then. (I know it's a super trivial thing to write: write a script that hits all our of most popular cached URLs frequently enough that it's highly unlikely that a user gets the cache miss. I don't like how that inflates our analytics/logs, so we should see if Akamai can do this.)

(4) Why does it take ~580 ms to fetch from the CDN a `.json file` that is 22.1 KB in size (4.5 KB gzip'd)? That's not a small number, especially since it blocks the loading of the homepage (if it's not in the offline `localStorage` request cache).

(5) We agree we should increase the `max-age`, we already serve the important public API responses from the CDN to avoid hitting Zamboni directly, and we cache all API responses in `localStorage` and refresh them asynchronously - but all these perf tricks are only masking an underlying problem with Zamboni/the webheads. Do we know why hitting Zamboni directly is taking so long? Servers? TLS/SSL? Nginx? Python? Django? Elasticsearch? MySQL? Redis? Which tools can we use to profile this and who can help?

––

Especially in regards to #4 and #5, I think we can all agree performance is important, so we need to address, profile, and measure.
1) Ok :)

2) To discuss in that bug, but the query string doesn't bother me, and at least we know it doesn't cause issues with caching / Vary.

3) I believe raising the TTL would severely improve cache hit ratio. If that's not enough then yes we can look into prefetching, either manually or by asking Akamai.

4), we need to make sure what the CDN is calling a cache hit. From my experience with other CDNs, what sometimes happen is that the PoP closest to you doesn't have the requested URL in its cache, so it asks another that sits further away but that should still answer faster than your origin. When that happens, you can sometimes lose a couple hundred milliseconds in the roundtrip. In any case I agree that ~ 580ms for a CDN response is a lot.

5), there are multiple factors. We know that we are not doing OSCP stapling (not sure if the CDN does), so at the very least depending on the browser there is some time wasted here. There might more TLS tricks (https://istlsfastyet.com/ has a good list) that we aren't doing yet as well. 
We also know that the feed endpoint has to make multiple ES queries, so we could try to improve this. That endpoint shouldn't make any MySQL or Redis requests. We could try investigating with the data from statsd.


I'm going to play with the TTL in this bug and see how much this improves the situation. Feel free to open up new bugs for other aspects :)
Assignee: nobody → mpillard
Priority: -- → P2
Target Milestone: --- → 2014-10-28
Status: NEW → ASSIGNED
Depends on: 1088329
IMO, this should be a tracking bug to improve the response time of the Feed (per the original description). And upping the `max-age` could've been a separate dependent bug on this one.

I've/others have filed several similar bugs about API response time, so I don't think this bug should actually get closed until we're satisfied with the response time.

I filed bug 1088329 to do in-depth profiling at the application, transport, and network layers.
Depends on: 1088361
Depends on: 1088362
Assignee: mpillard → nobody
Priority: P2 → --
Summary: Improve response time of homepage feed API endpoint → [tracking] Improve response time of homepage feed API endpoint
Target Milestone: 2014-10-28 → ---
Status: ASSIGNED → NEW
Depends on: 1088369
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.