Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 145889 reports (Page 19 of 2918)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X..........X.............] duration: 621.475842ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:31:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 621.529017ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.....X......] duration: 406.853304ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:31:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 406.879348ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.................X....] duration: 503.330295ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:31:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 503.371326ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X....X...................................X.......X......] duration: 482.108396ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:30:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.131645ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X...X.............X............X.......X...........] duration: 479.066342ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:30:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.106999ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..........X............] duration: 636.979622ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:30:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 637.004517ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.............X.....................X....] duration: 477.531066ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:30:38 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 477.581341ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.....X.....] duration: 456.316082ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:30:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 456.344355ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X....X...................................X.......X.....] duration: 560.598017ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:30:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 560.631616ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.................X...] duration: 761.641589ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:30:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 761.671421ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..........X...........] duration: 2.165636859s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:30:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.165662109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.............X.....................X...] duration: 462.563726ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:30:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 462.58715ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X...X.............X............X.......X..........] duration: 4.021250728s trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:30:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.021279122s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X....X...................................X.......X....] duration: 685.945223ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:30:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 686.090351ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.....X....] duration: 1.189034227s trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:30:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.189187333s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.....X...] duration: 21.130854731s trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:29:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 21.131019934s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..........X..........] duration: 25.947396162s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:29:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.947430711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.................X..] duration: 2.086206297s trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:29:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 2.086234885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.....X..] duration: 32.884318544s trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:29:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 32.884335112s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.............X.....................X..] duration: 1.219351133s trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:29:18 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.219372925s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X....X...................................X.......X...] duration: 1.042828691s trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:29:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.042852246s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X...X.............X............X.......X.........] duration: 796.340619ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:28:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 796.377304ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..........X.........] duration: 18.003518393s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:28:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.003637391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.................X.] duration: 1.546561827s trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:28:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 1.546586098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.................X] duration: 1m0.003107081s trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:28:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 1m0.003125903s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.....X.] duration: 24.622006437s trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:28:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 24.622027779s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..........X........] duration: 41.24608282s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:28:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.246179911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X....X...................................X.......X..] duration: 1.203196313s trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:28:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.203219411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.............X.....................X.] duration: 6.073518954s trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:28:14 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.073535331s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X...X.............X............X.......X........] duration: 4.468954066s trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:28:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.468964907s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.............X.....................X] duration: 30.026336814s trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:28:07 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.026356565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.....X] duration: 1m0.027524681s trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:28:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.027548298s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........X....X...................................X.......X.] duration: 754.301948ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:28:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 754.32137ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........X....X...................................X.......X] duration: 5.001135447s trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:27:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001155132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X...X.............X............X.......X.......] duration: 461.812143ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:26:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.843614ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X....X...................................X.......] duration: 736.810417ms trace: https://jaeger.dev.storj.io/trace/7e730421c2bf5cd3 FAIL (2025-10-31 21:26:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 736.835463ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X..........X.......] duration: 622.784558ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:26:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 622.82117ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 413.044264ms trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:26:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 413.059469ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.............X.....................] duration: 419.733404ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:26:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 419.838695ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X....X...................................X......] duration: 474.393545ms trace: https://jaeger.dev.storj.io/trace/7e730421c2bf5cd3 FAIL (2025-10-31 21:26:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.585644ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 402.539364ms trace: https://jaeger.dev.storj.io/trace/445049bafc385f46 FAIL (2025-10-31 21:26:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Oct 31 21:21:45.145651413 (33h42m15.809600402s) [upload=35035]: Starting upload Oct 31 21:21:45.145722499 (33h42m15.809671478s) [upload=35035]: Done starting upload Oct 31 21:21:45.145826139 (33h42m15.809775119s) [upload=35035]: Uploading segments... Oct 31 21:21:45.146035169 (33h42m15.809984149s) [upload=35035]: Got next segment. Inline: false Oct 31 21:21:45.146085119 (33h42m15.810034099s) [upload=35035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:21:45.146090534 (33h42m15.810039513s) [upload=35035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:21:45.175065231 (33h42m15.839014217s) [seg_pos={0 0}, upload=35035]: Begin upload segment... Oct 31 21:21:45.175123115 (33h42m15.839072098s) [seg_pos={0 0}, upload=35035]: Done begin upload segment. Oct 31 21:21:45.175180412 (33h42m15.839129395s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175189424 (33h42m15.839138404s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175260059 (33h42m15.839209041s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175699257 (33h42m15.839648244s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176115092 (33h42m15.840064077s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176555875 (33h42m15.840504854s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176971893 (33h42m15.840920873s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.177678646 (33h42m15.841627628s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.178186917 (33h42m15.842135889s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180863141 (33h42m15.844812123s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180996018 (33h42m15.844945004s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.181751937 (33h42m15.845700926s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.182437363 (33h42m15.846386353s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.183063177 (33h42m15.84701216s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.184526693 (33h42m15.848475673s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185052533 (33h42m15.849001512s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185337329 (33h42m15.849286296s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.186151456 (33h42m15.850100442s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.255741810 (33h42m15.919690796s) [upload=35035]: Next returned nil segment Oct 31 21:21:45.255755669 (33h42m15.91970465s) [upload=35035]: Waiting for error group managing segments... Oct 31 21:21:45.281261953 (33h42m15.945210932s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.281365241 (33h42m15.945314236s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287923484 (33h42m15.95187247s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287939415 (33h42m15.951888402s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.298744521 (33h42m15.962693508s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.301331841 (33h42m15.96528083s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.303954834 (33h42m15.967903813s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:31.494039004 (33h43m2.157987983s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:33.548973577 (33h43m4.212922558s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:39.893068537 (33h43m10.557017516s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:41.428154396 (33h43m12.092103375s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.747689188 (33h43m15.411638168s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.806294014 (33h43m15.470243s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:45.101926872 (33h43m15.765875858s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm, address:45.91.116.5:4327): protocol: read tcp 159.223.143.39:43818->45.91.116.5:4327: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.101996547 (33h43m15.765945536s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: rpc: dial tcp 77.90.31.11:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102228470 (33h43m15.76617745s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: write tcp 159.223.143.39:46056->77.90.31.21:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102287458 (33h43m15.766236445s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection Oct 31 21:22:45.102659278 (33h43m15.766608268s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 159.223.143.39:47504->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102701510 (33h43m15.766650478s) [seg_pos={0 0}, upload=35035]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 31 21:22:45.121685275 (33h43m15.785634244s) [upload=35035]: Done uploading segments... Oct 31 21:22:45.121721395 (33h43m15.785670378s) [upload=35035]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 402.555127ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X...X.............X............X.......X......] duration: 444.538035ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:26:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 444.565339ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X....X...................................X.....] duration: 445.632959ms trace: https://jaeger.dev.storj.io/trace/7e730421c2bf5cd3 FAIL (2025-10-31 21:25:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 445.667375ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 448.7897ms trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:25:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 448.888668ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..........X......] duration: 589.143977ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:25:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 589.285827ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 430.442231ms trace: https://jaeger.dev.storj.io/trace/445049bafc385f46 FAIL (2025-10-31 21:25:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Oct 31 21:21:45.145651413 (33h42m15.809600402s) [upload=35035]: Starting upload Oct 31 21:21:45.145722499 (33h42m15.809671478s) [upload=35035]: Done starting upload Oct 31 21:21:45.145826139 (33h42m15.809775119s) [upload=35035]: Uploading segments... Oct 31 21:21:45.146035169 (33h42m15.809984149s) [upload=35035]: Got next segment. Inline: false Oct 31 21:21:45.146085119 (33h42m15.810034099s) [upload=35035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:21:45.146090534 (33h42m15.810039513s) [upload=35035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:21:45.175065231 (33h42m15.839014217s) [seg_pos={0 0}, upload=35035]: Begin upload segment... Oct 31 21:21:45.175123115 (33h42m15.839072098s) [seg_pos={0 0}, upload=35035]: Done begin upload segment. Oct 31 21:21:45.175180412 (33h42m15.839129395s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175189424 (33h42m15.839138404s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175260059 (33h42m15.839209041s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.175699257 (33h42m15.839648244s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176115092 (33h42m15.840064077s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176555875 (33h42m15.840504854s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.176971893 (33h42m15.840920873s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.177678646 (33h42m15.841627628s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.178186917 (33h42m15.842135889s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180863141 (33h42m15.844812123s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.180996018 (33h42m15.844945004s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.181751937 (33h42m15.845700926s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.182437363 (33h42m15.846386353s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.183063177 (33h42m15.84701216s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.184526693 (33h42m15.848475673s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185052533 (33h42m15.849001512s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.185337329 (33h42m15.849286296s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.186151456 (33h42m15.850100442s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Uploading piece... Oct 31 21:21:45.255741810 (33h42m15.919690796s) [upload=35035]: Next returned nil segment Oct 31 21:21:45.255755669 (33h42m15.91970465s) [upload=35035]: Waiting for error group managing segments... Oct 31 21:21:45.281261953 (33h42m15.945210932s) [noise=true , address=145.79.227.38:4003 , piece_id=2VGZDNK6WP3B6FUIFLAZDIO4S2A3FSL6OMOSON5DKLXT3RYZWDIQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.281365241 (33h42m15.945314236s) [noise=true , address=145.79.227.33:4897 , piece_id=LKGYSEPEDZ5BUSLHHBCL6GWPXUM6USAIK6GG7DOUQLG6KSDFUDJQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287923484 (33h42m15.95187247s) [noise=true , address=145.79.227.33:4904 , piece_id=GL3ZIRCGGZTPQU3ZDKQ7AJ6UWBRA763PBI6OVM645FLFF6VNGQFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.287939415 (33h42m15.951888402s) [noise=true , address=145.79.227.43:4192 , piece_id=YOLC4CM4CI7FQ2GKGRYKQIG3VSWAXC3FC65V7NPGPY7UMNROORYA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.298744521 (33h42m15.962693508s) [noise=true , address=145.79.227.4:4216 , piece_id=UD6KN3NV3473DTXLUOK7FBFP4V4OVRDTZST3BIHCRUHBQNKKYILQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.301331841 (33h42m15.96528083s) [noise=true , address=145.79.227.5:4226 , piece_id=F6GRFWVU6BFXG7ONHDF7QJMVSHNANRZTD7DD2KXNUGGFWFOW4OBQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:21:45.303954834 (33h42m15.967903813s) [noise=true , address=145.79.227.44:4227 , piece_id=BXQD3PX4QDVEV6WTTHE3VHKMPSL55JX3FBVDJYA3SZECIYXSJKLA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:31.494039004 (33h43m2.157987983s) [noise=true , address=45.91.116.10:4092 , piece_id=DCGYEF7BNKGZPY2SS5ONEPHMAHE4W7TVN5TFOYEXTGWABXC44Y5Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:33.548973577 (33h43m4.212922558s) [noise=true , address=77.90.31.14:4688 , piece_id=3MEUY3PNYEBNIAMSVVHRQY3YGHDJDELCMWST6EQLRYB7NF5QFQ3Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:39.893068537 (33h43m10.557017516s) [noise=true , address=77.90.31.17:4779 , piece_id=ZLCEI3JFPKLCW3MGPFDOQP7Q4X42WWNOCK5CXLHU6T2NC6DGJLFQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:41.428154396 (33h43m12.092103375s) [noise=true , address=45.91.116.9:4268 , piece_id=SZU4ICBBCGG3CXWK5QWRGXRTKHYMFQB4ODSWFLCCXRMP5KE65O2Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.747689188 (33h43m15.411638168s) [noise=true , address=45.91.116.14:4208 , piece_id=XUF3AAXVX7JMXRUTHZJIBREUWQNFJWTS5WXQOST476NQWHUHTGXA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:44.806294014 (33h43m15.470243s) [noise=true , address=45.91.116.13:4180 , piece_id=BSV7COCRYF3EWQJIDLNVKX5EZGLT5ISZUQZ66J4XX7SSRTJYLCVA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: <nil> Oct 31 21:22:45.101926872 (33h43m15.765875858s) [noise=true , address=45.91.116.5:4327 , piece_id=WJFXO2FN4NLTAX5NEVSUUPSEFKPGUYL55OYML66SVUAKG3YVUZSA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12HY8UAuJNnivgSkxwNfFETRLrFFCtAnGsjkGHVvby1KCycDzSm, address:45.91.116.5:4327): protocol: read tcp 159.223.143.39:43818->45.91.116.5:4327: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.101996547 (33h43m15.765945536s) [noise=true , address=77.90.31.11:4570 , piece_id=ZNGBHQ4QPB3Q4ZAHNVMASKN6WQ55H455DRXR5EO66K4QY32BPHWQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, address:77.90.31.11:4570): protocol: rpc: dial tcp 77.90.31.11:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102228470 (33h43m15.76617745s) [noise=true , address=77.90.31.21:4905 , piece_id=TPUZG4Q4XWWKBY7OZS7WMU5MMVHCA3RXGCJUATB6OCA3D5QQK6IA, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: write tcp 159.223.143.39:46056->77.90.31.21:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102287458 (33h43m15.766236445s) [noise=true , address=77.90.31.15:4718 , piece_id=FRCYEQCKVL5XIOWWNOJJFUY5YRSO3ZFUBCOPNVPMVFIAYM6MFRYQ, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49702->77.90.31.15:4718: use of closed network connection Oct 31 21:22:45.102659278 (33h43m15.766608268s) [noise=true , address=45.91.116.6:4983 , piece_id=YAUSEWKPRIL3KKPY7NW4XD47RY2QGKWIZF47D4O3IU26QKPES53Q, seg_pos={0 0}, upload=35035]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 159.223.143.39:47504->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:22:45.102701510 (33h43m15.766650478s) [seg_pos={0 0}, upload=35035]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 31 21:22:45.121685275 (33h43m15.785634244s) [upload=35035]: Done uploading segments... Oct 31 21:22:45.121721395 (33h43m15.785670378s) [upload=35035]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 430.467686ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X....X...................................X....] duration: 550.910007ms trace: https://jaeger.dev.storj.io/trace/7e730421c2bf5cd3 FAIL (2025-10-31 21:25:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 551.019619ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.............X....................] duration: 448.273243ms trace: https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335 FAIL (2025-10-31 21:25:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 8]: Uploading segments... Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment... Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment. Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece... Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments... Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil> Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments... Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 448.301365ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X...X.............X............X.......X.....] duration: 577.595458ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:25:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 577.626452ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 575.958864ms trace: https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d FAIL (2025-10-31 21:24:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece... Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments... Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil> Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments... Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 575.985114ms core uplink us-select-1 us1-placement-20

1
...
17
18
19
20
21
...
2918