Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029747561s FAIL (2025-07-26 22:16:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 207 (23m17.161303601s) [noise=true , address=79.127.219.38:4003 , piece_id=SNEXFG5WYJKZQJKA7LO66H4QWXIFUDT7F2OA6YG7Q3GJ66AIH3QA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.381141949 (23m17.162075338s) [noise=true , address=104.36.82.52:4089 , piece_id=IKUESTCMKMPXG5RILFLEXMKALF3U3FAWX5YAQLCL7P3Z2UPPXTOQ, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.381747090 (23m17.162680477s) [noise=true , address=104.36.82.20:4450 , piece_id=GB7FD4JA6ALNHUU456H6DPRUWNFDOXVMQXXRI3TLVIQIUFNTIOPQ, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.383633106 (23m17.164566488s) [noise=true , address=104.36.82.56:4264 , piece_id=SYT2VME2PVBBDOKT7KU4XMCXCLYB7E6BLHDAZ7BGPCAME2X5RGGQ, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.384385318 (23m17.165318707s) [noise=true , address=104.36.82.53:4122 , piece_id=HTOBANSLEJWB3L2FPO2GJIEXKY72AO3CAWX7FP46CJJRQMMPRGSA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.384819879 (23m17.165753261s) [noise=true , address=104.36.82.18:4400 , piece_id=WSGJRSWA64GZ6YZ4RG3XMXGDWG6TFO4ABFQPFT2NI3ME5HTNWFPA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.385609781 (23m17.166543172s) [noise=true , address=79.127.219.51:4212 , piece_id=ZKMHBMSFWI3MU426VOIZJAFC7RHMHNI3U4PVM6QZVXBPH2TAPYFQ, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.386549424 (23m17.167482813s) [noise=true , address=77.90.31.10:4533 , piece_id=5UNYLWAYXB6JUP3N5AJFX5WZO5F6AW6ZEJQDMHYYEZ4FQP2JQUWA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.387624314 (23m17.168557702s) [noise=true , address=104.36.82.14:4243 , piece_id=VW54S5OCLNCHZ7QBJDOMUMHN5242D5IUTCGUHFT3JRBQGR5QVIRQ, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.388751749 (23m17.169685137s) [noise=true , address=104.36.82.49:4978 , piece_id=C7ZCMIBVWBWFVNQNICLNGFJ4YWKTM5FQISUJVLPTQZ5LR4465EGA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.389242140 (23m17.170175528s) [noise=true , address=79.127.205.246:4287 , piece_id=SCAWLDCW6VY4ZBNQLQEPHIJXFEHLKTCH65FISV2AC2M4YTGLEPWA, seg_pos={0 0}, upload=363]: Uploading piece... Jul 26 22:15:01.535167746 (23m17.316101133s) [upload=363]: Next returned nil segment Jul 26 22:15:01.535191331 (23m17.31612472s) [upload=363]: Waiting for error group managing segments... Jul 26 22:15:01.588605141 (23m17.369538532s) [noise=true , address=77.90.31.10:4533 , piece_id=5UNYLWAYXB6JUP3N5AJFX5WZO5F6AW6ZEJQDMHYYEZ4FQP2JQUWA, seg_pos={0 0}, upload=363]: Done uploading piece. err: <nil> Jul 26 22:15:01.657193776 (23m17.43812714s) [noise=true , address=79.127.219.38:4003 , piece_id=SNEXFG5WYJKZQJKA7LO66H4QWXIFUDT7F2OA6YG7Q3GJ66AIH3QA, seg_pos={0 0}, upload=363]: Done uploading piece. err: <nil> Jul 26 22:15:01.834852009 (23m17.615785401s) [noise=true , address=79.127.219.51:4212 , piece_id=ZKMHBMSFWI3MU426VOIZJAFC7RHMHNI3U4PVM6QZVXBPH2TAPYFQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: <nil> Jul 26 22:15:02.284727994 (23m18.065661375s) [noise=true , address=79.127.205.246:4287 , piece_id=SCAWLDCW6VY4ZBNQLQEPHIJXFEHLKTCH65FISV2AC2M4YTGLEPWA, seg_pos={0 0}, upload=363]: Done uploading piece. err: <nil> Jul 26 22:15:03.065418763 (23m18.846352146s) [noise=true , address=79.127.201.215:4356 , piece_id=RSOVA25MF5G6U2HB32SUFDJPL7KUFE6KGTDB4ESYKO363AQ5QL4A, seg_pos={0 0}, upload=363]: Done uploading piece. err: <nil> Jul 26 22:16:01.278094005 (24m17.059027385s) [noise=true , address=104.36.82.20:4471 , piece_id=HL3ESI753MHCYS5UKGG7GOC62ATLVVNIXG7T3WSC45Y5WQFAPRNA, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12DMRnzj9hpP9tRn5DUz9WcU3N4jEZ3CZ9saVTFa1tefjDNs72r, address:104.36.82.20:4471): protocol: write tcp 159.223.143.39:57490->104.36.82.20:4471: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.278398771 (24m17.059332147s) [noise=true , address=104.36.82.20:4450 , piece_id=GB7FD4JA6ALNHUU456H6DPRUWNFDOXVMQXXRI3TLVIQIUFNTIOPQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12UeL1LySKQySa8wjvG4FBs69cE1iWkW4Diz83nu1PxBCKxyvki, address:104.36.82.20:4450): protocol: write tcp 159.223.143.39:57250->104.36.82.20:4450: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.278755728 (24m17.059689106s) [noise=true , address=104.36.82.50:4013 , piece_id=U4X5MP4H7VTYTLXJNCHOEOF6NEJGNOZCSH7HCS4SGYIMIQBX2AAQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:104.36.82.50:4013): protocol: write tcp 159.223.143.39:40656->104.36.82.50:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.278845053 (24m17.059778433s) [noise=true , address=104.36.82.22:4531 , piece_id=OM3BCN4DZGLNN6OE6IVHF2BUILUDTIOFIAQFTHUF4WEETVJMWKRQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12jGenn3Fh2VB36jVPzyy2LxJgbLraXxL2yobw9tsBJLSDPJycN, address:104.36.82.22:4531): protocol: write tcp 159.223.143.39:52914->104.36.82.22:4531: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.278937528 (24m17.059870906s) [noise=true , address=104.36.82.52:4089 , piece_id=IKUESTCMKMPXG5RILFLEXMKALF3U3FAWX5YAQLCL7P3Z2UPPXTOQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: write tcp 159.223.143.39:35100->104.36.82.52:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.278994899 (24m17.059928266s) [noise=true , address=104.36.82.53:4122 , piece_id=HTOBANSLEJWB3L2FPO2GJIEXKY72AO3CAWX7FP46CJJRQMMPRGSA, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: write tcp 159.223.143.39:43732->104.36.82.53:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279108733 (24m17.060042099s) [noise=true , address=104.36.82.51:4072 , piece_id=KH7NH6WNGKLYZCEOKZL6XRGWPF3NJ5IVCL7VSKM4MGGNW2RBEGSQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne, address:104.36.82.51:4072): protocol: write tcp 159.223.143.39:41306->104.36.82.51:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279178294 (24m17.060111669s) [noise=true , address=104.36.82.55:4196 , piece_id=TZ5D22G6OGPLQ6MP4QPCFU5F2L2KWDO6OXD3WGTNWNK463RENNKQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: write tcp 159.223.143.39:49606->104.36.82.55:4196: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279243926 (24m17.060177311s) [noise=true , address=104.36.82.14:4243 , piece_id=VW54S5OCLNCHZ7QBJDOMUMHN5242D5IUTCGUHFT3JRBQGR5QVIRQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:1FAn2HGJpXP5dfbYfL8PkxCfrBM3SHJVCASQMSy1kaRqLhyiRv, address:104.36.82.14:4243): protocol: write tcp 159.223.143.39:36398->104.36.82.14:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279270304 (24m17.060203674s) [noise=true , address=104.36.82.46:4515 , piece_id=UQMP35ZOJTLY6M7UU3SAKNT65O5B4XY5UQTNJQOS4BLREX4IPJOA, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:52702->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279336827 (24m17.060270318s) [noise=true , address=104.36.82.56:4264 , piece_id=SYT2VME2PVBBDOKT7KU4XMCXCLYB7E6BLHDAZ7BGPCAME2X5RGGQ, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: write tcp 159.223.143.39:33982->104.36.82.56:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279376952 (24m17.06031032s) [noise=true , address=104.36.82.49:4978 , piece_id=C7ZCMIBVWBWFVNQNICLNGFJ4YWKTM5FQISUJVLPTQZ5LR4465EGA, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:104.36.82.49:4978): protocol: write tcp 159.223.143.39:50782->104.36.82.49:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279430818 (24m17.060364186s) [noise=true , address=104.36.82.18:4400 , piece_id=WSGJRSWA64GZ6YZ4RG3XMXGDWG6TFO4ABFQPFT2NI3ME5HTNWFPA, seg_pos={0 0}, upload=363]: Done uploading piece. err: ecclient: upload failed (node:12UxA6DMP2dr2T8o352iih2D6UeEeGQBMm1SgPbHYHuWaSFdyYv, address:104.36.82.18:4400): protocol: write tcp 159.223.143.39:42226->104.36.82.18:4400: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:16:01.279468321 (24m17.060401716s) [seg_pos={0 0}, upload=363]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:16:01.303336944 (24m17.084270327s) [upload=363]: Done uploading segments... Jul 26 22:16:01.303369884 (24m17.084303262s) [upload=363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.029764521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [............................X..X........XXX.XXX.X.XX..X..X..] duration: 550.49436ms FAIL (2025-07-26 22:16:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 550.5572ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 8.853640355s FAIL (2025-07-26 22:15:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.85366664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............................XX..XXX..X..XX..X..XX.X....X...] duration: 516.6814ms FAIL (2025-07-26 22:15:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tocol: rpc: dial tcp 104.36.82.15:4257: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864074530 (20m15.64500791s) [noise=true , address=104.36.82.4:4802 , piece_id=VA26G7K6XQFH4PHZCJ2XWTGOD4GD5JR6DPVR35KCMLMWP4TGBXMA, seg_pos={0 0}, upload=320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:35088->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864116163 (20m15.645049543s) [seg_pos={0 0}, upload=320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:59.864361342 (20m15.645294726s) [upload=320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:59.864371913 (20m15.645305299s) [upload=320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:59.917691593 (20m15.698624962s) [upload=320]: Done uploading segments... Jul 26 22:11:59.917714191 (20m15.698647562s) [upload=320]: Upload finished. err: <nil> Jul 26 22:12:53.913781892 (21m9.69471528s) [upload=334]: Starting upload Jul 26 22:12:53.913910566 (21m9.694843946s) [upload=334]: Done starting upload Jul 26 22:12:53.914068111 (21m9.695001479s) [upload=334]: Uploading segments... Jul 26 22:12:53.914418778 (21m9.695352164s) [upload=334]: Got next segment. Inline: false Jul 26 22:12:53.914489104 (21m9.695422468s) [upload=334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:53.914499080 (21m9.695432464s) [upload=334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:53.964779415 (21m9.745712792s) [seg_pos={0 0}, upload=334]: Begin upload segment... Jul 26 22:12:53.964834832 (21m9.745768199s) [seg_pos={0 0}, upload=334]: Done begin upload segment. Jul 26 22:12:53.964861055 (21m9.745794419s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.964889604 (21m9.745822981s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965340371 (21m9.746273751s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965731230 (21m9.746664613s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966122432 (21m9.747055812s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966494352 (21m9.747427739s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966826509 (21m9.747759885s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967213838 (21m9.748147227s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967583090 (21m9.748516472s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969295495 (21m9.75022888s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969752444 (21m9.750685812s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970073562 (21m9.75100693s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970432919 (21m9.751366305s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.971368537 (21m9.752301923s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.972505536 (21m9.753438924s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.973901396 (21m9.754834793s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:54.096343157 (21m9.877276542s) [upload=334]: Next returned nil segment Jul 26 22:12:54.096365409 (21m9.877298804s) [upload=334]: Waiting for error group managing segments... Jul 26 22:12:54.102545693 (21m9.883479076s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103199713 (21m9.884133094s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103330731 (21m9.884264101s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103766692 (21m9.884700057s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104301838 (21m9.885235206s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104452646 (21m9.885386024s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104983988 (21m9.885917367s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105066863 (21m9.886000233s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105555466 (21m9.886488829s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105740907 (21m9.886674273s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108750596 (21m9.889683966s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108767895 (21m9.889701273s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:13:23.875895875 (21m39.656829258s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875916089 (21m39.656849455s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875996243 (21m39.656929626s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:33108->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876011671 (21m39.656945039s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:47806->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876035419 (21m39.656968803s) [seg_pos={0 0}, upload=334]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:13:23.902382513 (21m39.683315882s) [upload=334]: Done uploading segments... Jul 26 22:13:23.902433922 (21m39.683367306s) [upload=334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 516.899753ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............................X..X........XXX.XXX.X.XX..X..X.] duration: 487.747512ms FAIL (2025-07-26 22:15:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 487.941243ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [..............................X..X........XXX.XXX.X.XX..X..X] duration: 30.048257977s FAIL (2025-07-26 22:15:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.04829145s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.00104872s FAIL (2025-07-26 22:15:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001077568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029585394s FAIL (2025-07-26 22:15:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 2 , piece_id=QJVOO3LHRH2IXFNRAE6RNSARSRUFK6SH76UMZJ3RIRVAT264IINA, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.243680224 (22m17.024613608s) [noise=true , address=79.127.205.235:4682 , piece_id=LMXRIY4YQ6OQQYVTP722DDQP55XIUYN67YCDHCTCPQXTMF5GWRRQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.243932574 (22m17.024865942s) [noise=true , address=77.90.31.11:4579 , piece_id=5D5OF6UU25BJQA5FKWQJNQJOYTNNPYU56BYTHZVAHDAC5KUKMLFQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.244087833 (22m17.02502121s) [noise=true , address=77.90.31.18:4822 , piece_id=JXKQPWVTHAYXEONUJHP3PHD7O324PJUJUAEL6OF37FKDNXE6TE6A, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.244643513 (22m17.025576892s) [noise=true , address=79.127.201.215:4343 , piece_id=E33ZGZG4MW36CZE5IDGQJ3EXLK5DGBKO7I7VEQOZVCNNDOWA7GDA, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.244683583 (22m17.02561697s) [noise=true , address=45.91.116.14:4222 , piece_id=OK3B2RGFDUI42ETOQV3U5EQXVFUWBTPZK7REIPBRMPAJZSQZJLHQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.244971930 (22m17.025905309s) [noise=true , address=104.36.82.13:4207 , piece_id=A37CBRYK2XUUBPGHZHZVFLNGABHNGKBTWPOSIR2WEOJIVE5AUXRA, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.245281092 (22m17.026214479s) [noise=true , address=104.36.82.46:4489 , piece_id=MQD4VSRFLEYUNJU2ON34M5X3MLYBZ24LO7UZ3MB5CIMQWGQOLLFQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.245326951 (22m17.026260331s) [noise=true , address=104.36.82.53:4143 , piece_id=FOOCWCIO5C7TPZYLNJJFN6XGPWOG3OSE577PXHKUAO2BQ7T7ONXQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.245664939 (22m17.026598318s) [noise=true , address=104.36.82.22:4536 , piece_id=IQGR6B6JAHA7BUVMKKNL6LPIWZBCZHRVFB52E7WOXJCPOL677VNQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.245925159 (22m17.026858543s) [noise=true , address=104.36.82.55:4196 , piece_id=XLLCTG7QI2GIQZWYVUF7KBRV6VBVQGV2KLRC7ZM7G2BGFXU4DKPA, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.246035744 (22m17.026969122s) [noise=true , address=104.36.82.47:4528 , piece_id=K7BHBZDPGFXERLSC4YFAVDS36AWL2WMGDXZGDVE4ITTK6QYG3JLQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.246558793 (22m17.027492185s) [noise=true , address=104.36.82.56:4243 , piece_id=TPPYUDVKOTAMAYRYLOVMOKWXTBIJDWJBXK5WYGQR3MGDU6HAHKRQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.247150043 (22m17.028083432s) [noise=true , address=77.90.31.16:4760 , piece_id=EP5RSDB75UZM6FDRJ7KY5RH5BU3UR74S6ED6VRURVIQPS3AK5I3A, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.247692056 (22m17.028625446s) [noise=true , address=79.127.205.234:4619 , piece_id=PBBI2Z5KWL2G2L6SYCFNKPB536DNWRTN45EKVZFXXX3RKITYY5DQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.248256110 (22m17.029189499s) [noise=true , address=104.36.82.44:4887 , piece_id=ULDZEFV2PWTG7VDMTVF6Q3ZAPDJ3TBIJU3EY37JY2NQFBGJMEMZQ, seg_pos={0 0}, upload=348]: Uploading piece... Jul 26 22:14:01.327424771 (22m17.108358198s) [upload=348]: Next returned nil segment Jul 26 22:14:01.327450970 (22m17.108384357s) [upload=348]: Waiting for error group managing segments... Jul 26 22:14:01.841386055 (22m17.622319433s) [noise=true , address=79.127.205.235:4682 , piece_id=LMXRIY4YQ6OQQYVTP722DDQP55XIUYN67YCDHCTCPQXTMF5GWRRQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:01.943651445 (22m17.724584815s) [noise=true , address=79.127.205.234:4619 , piece_id=PBBI2Z5KWL2G2L6SYCFNKPB536DNWRTN45EKVZFXXX3RKITYY5DQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:08.999820844 (22m24.780754242s) [noise=true , address=77.90.31.16:4760 , piece_id=EP5RSDB75UZM6FDRJ7KY5RH5BU3UR74S6ED6VRURVIQPS3AK5I3A, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:09.764776235 (22m25.545709601s) [noise=true , address=45.91.116.14:4222 , piece_id=OK3B2RGFDUI42ETOQV3U5EQXVFUWBTPZK7REIPBRMPAJZSQZJLHQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:10.344858287 (22m26.125791893s) [noise=true , address=79.127.201.215:4343 , piece_id=E33ZGZG4MW36CZE5IDGQJ3EXLK5DGBKO7I7VEQOZVCNNDOWA7GDA, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:13.090150330 (22m28.871083691s) [noise=true , address=77.90.31.18:4822 , piece_id=JXKQPWVTHAYXEONUJHP3PHD7O324PJUJUAEL6OF37FKDNXE6TE6A, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:14:14.048748563 (22m29.829681932s) [noise=true , address=64.49.13.138:4482 , piece_id=QJVOO3LHRH2IXFNRAE6RNSARSRUFK6SH76UMZJ3RIRVAT264IINA, seg_pos={0 0}, upload=348]: Done uploading piece. err: <nil> Jul 26 22:15:01.141100547 (23m16.92203393s) [noise=true , address=104.36.82.2:4745 , piece_id=KTXAGLEJWDHSO5OCOG3XTNZ2YQER4VDG6OPPZQ7TEK2LYDDSB42A, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:1H9sLaFPDPZxKvwXho76wrqPiFe9yu87Ue9BT4ip92qfLmjbnF, address:104.36.82.2:4745): protocol: rpc: dial tcp 104.36.82.2:4745: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141424241 (23m16.922357622s) [noise=true , address=104.36.82.50:4025 , piece_id=O3RISI5REZAPP3YZH5XHM7BLTRYML3QQOVXXCLSHIKKJ5YXZYNTA, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:104.36.82.50:4025): protocol: rpc: dial tcp 104.36.82.50:4025: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141498271 (23m16.922431635s) [noise=true , address=104.36.82.13:4207 , piece_id=A37CBRYK2XUUBPGHZHZVFLNGABHNGKBTWPOSIR2WEOJIVE5AUXRA, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141587075 (23m16.922520453s) [noise=true , address=104.36.82.46:4489 , piece_id=MQD4VSRFLEYUNJU2ON34M5X3MLYBZ24LO7UZ3MB5CIMQWGQOLLFQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: rpc: dial tcp 104.36.82.46:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141653846 (23m16.922587224s) [noise=true , address=104.36.82.22:4536 , piece_id=IQGR6B6JAHA7BUVMKKNL6LPIWZBCZHRVFB52E7WOXJCPOL677VNQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12C2G1QGN37Wu55dBAxLiYWYAXmGapzjWUZUCKgCUbcjXcaRFmt, address:104.36.82.22:4536): protocol: write tcp 159.223.143.39:43194->104.36.82.22:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141713617 (23m16.922646994s) [noise=true , address=104.36.82.47:4528 , piece_id=K7BHBZDPGFXERLSC4YFAVDS36AWL2WMGDXZGDVE4ITTK6QYG3JLQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: rpc: dial tcp 104.36.82.47:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141735749 (23m16.922669127s) [noise=true , address=77.90.31.11:4579 , piece_id=5D5OF6UU25BJQA5FKWQJNQJOYTNNPYU56BYTHZVAHDAC5KUKMLFQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:15:01.141817910 (23m16.922751288s) [noise=true , address=104.36.82.56:4243 , piece_id=TPPYUDVKOTAMAYRYLOVMOKWXTBIJDWJBXK5WYGQR3MGDU6HAHKRQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: rpc: dial tcp 104.36.82.56:4243: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.141931334 (23m16.922864717s) [noise=true , address=104.36.82.53:4143 , piece_id=FOOCWCIO5C7TPZYLNJJFN6XGPWOG3OSE577PXHKUAO2BQ7T7ONXQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: rpc: dial tcp 104.36.82.53:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.142042923 (23m16.922976301s) [noise=true , address=104.36.82.55:4196 , piece_id=XLLCTG7QI2GIQZWYVUF7KBRV6VBVQGV2KLRC7ZM7G2BGFXU4DKPA, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: rpc: dial tcp 104.36.82.55:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.142091658 (23m16.923025036s) [noise=true , address=104.36.82.44:4887 , piece_id=ULDZEFV2PWTG7VDMTVF6Q3ZAPDJ3TBIJU3EY37JY2NQFBGJMEMZQ, seg_pos={0 0}, upload=348]: Done uploading piece. err: ecclient: upload failed (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H, address:104.36.82.44:4887): protocol: write tcp 159.223.143.39:38072->104.36.82.44:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:15:01.142135504 (23m16.923068872s) [seg_pos={0 0}, upload=348]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:15:01.168270238 (23m16.949203616s) [upload=348]: Done uploading segments... Jul 26 22:15:01.168309637 (23m16.949243017s) [upload=348]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.029614945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..............................XX..XXX..X..XX..X..XX.X....X..] duration: 434.521406ms FAIL (2025-07-26 22:14:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tocol: rpc: dial tcp 104.36.82.15:4257: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864074530 (20m15.64500791s) [noise=true , address=104.36.82.4:4802 , piece_id=VA26G7K6XQFH4PHZCJ2XWTGOD4GD5JR6DPVR35KCMLMWP4TGBXMA, seg_pos={0 0}, upload=320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:35088->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864116163 (20m15.645049543s) [seg_pos={0 0}, upload=320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:59.864361342 (20m15.645294726s) [upload=320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:59.864371913 (20m15.645305299s) [upload=320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:59.917691593 (20m15.698624962s) [upload=320]: Done uploading segments... Jul 26 22:11:59.917714191 (20m15.698647562s) [upload=320]: Upload finished. err: <nil> Jul 26 22:12:53.913781892 (21m9.69471528s) [upload=334]: Starting upload Jul 26 22:12:53.913910566 (21m9.694843946s) [upload=334]: Done starting upload Jul 26 22:12:53.914068111 (21m9.695001479s) [upload=334]: Uploading segments... Jul 26 22:12:53.914418778 (21m9.695352164s) [upload=334]: Got next segment. Inline: false Jul 26 22:12:53.914489104 (21m9.695422468s) [upload=334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:53.914499080 (21m9.695432464s) [upload=334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:53.964779415 (21m9.745712792s) [seg_pos={0 0}, upload=334]: Begin upload segment... Jul 26 22:12:53.964834832 (21m9.745768199s) [seg_pos={0 0}, upload=334]: Done begin upload segment. Jul 26 22:12:53.964861055 (21m9.745794419s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.964889604 (21m9.745822981s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965340371 (21m9.746273751s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965731230 (21m9.746664613s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966122432 (21m9.747055812s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966494352 (21m9.747427739s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966826509 (21m9.747759885s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967213838 (21m9.748147227s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967583090 (21m9.748516472s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969295495 (21m9.75022888s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969752444 (21m9.750685812s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970073562 (21m9.75100693s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970432919 (21m9.751366305s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.971368537 (21m9.752301923s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.972505536 (21m9.753438924s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.973901396 (21m9.754834793s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:54.096343157 (21m9.877276542s) [upload=334]: Next returned nil segment Jul 26 22:12:54.096365409 (21m9.877298804s) [upload=334]: Waiting for error group managing segments... Jul 26 22:12:54.102545693 (21m9.883479076s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103199713 (21m9.884133094s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103330731 (21m9.884264101s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103766692 (21m9.884700057s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104301838 (21m9.885235206s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104452646 (21m9.885386024s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104983988 (21m9.885917367s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105066863 (21m9.886000233s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105555466 (21m9.886488829s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105740907 (21m9.886674273s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108750596 (21m9.889683966s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108767895 (21m9.889701273s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:13:23.875895875 (21m39.656829258s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875916089 (21m39.656849455s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875996243 (21m39.656929626s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:33108->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876011671 (21m39.656945039s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:47806->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876035419 (21m39.656968803s) [seg_pos={0 0}, upload=334]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:13:23.902382513 (21m39.683315882s) [upload=334]: Done uploading segments... Jul 26 22:13:23.902433922 (21m39.683367306s) [upload=334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.553345ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 477.378606ms FAIL (2025-07-26 22:14:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.426298ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [...............................X..X........XXX.XXX.X.XX..X..] duration: 485.606987ms FAIL (2025-07-26 22:14:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 485.64932ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 516.806174ms FAIL (2025-07-26 22:14:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 516.828625ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 472.441641ms FAIL (2025-07-26 22:13:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 472.470558ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.0332824s FAIL (2025-07-26 22:13:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 48496s) [noise=true , address=104.36.82.42:4809 , piece_id=WLJONQZYSEDY4OGJ5KCQPF7JUWJPRWZTG7NUAQY4JB7D3R3IGGTA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.885564432 (21m4.666497818s) [noise=true , address=79.127.201.215:4350 , piece_id=Y7EARFRFLNNZQMSCSL27OJ5P6DH42DJBZ5TWO7G7QUPKQKB5P2RQ, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.885949517 (21m4.666882903s) [noise=true , address=104.36.82.3:4783 , piece_id=MGNS6XRLLLKUEWGJLPM55KEV4F73LLTRXNH7S7ZKHKXIVCLPFGWA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.886238666 (21m4.667172051s) [noise=true , address=77.90.31.11:4579 , piece_id=MUZRLA67GHCABWG7NOTN2WILNUEAFHEO2JCZFURG6H4P3DXCE4IA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.886331816 (21m4.667265202s) [noise=true , address=104.36.82.56:4251 , piece_id=JFJ33CH34YKRVXF2T3OWBX5NXU6NHCZMA7D7G6NITS4QR4A3OWTA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.886643846 (21m4.667577276s) [noise=true , address=79.127.205.234:4612 , piece_id=OPG2RRVGEYTEI733WQIX5B2V7RETEGR6UH53DNAPIAUJED5GFJXA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.887069399 (21m4.668002784s) [noise=true , address=64.49.13.169:4716 , piece_id=2SBENU3JIVLK7JP77U2DNPZ5J2G5DR4TXMC5DQVPTC74MHXH7PEA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.887462789 (21m4.668396167s) [noise=true , address=104.36.82.45:4916 , piece_id=XOUNTRQKZFOHHP23CYGUVP7N5V4MHZLKKHVHMNQDLIVHELWPYL3A, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.887751055 (21m4.668684433s) [noise=true , address=79.127.205.245:4231 , piece_id=G7OUM6FPKEKC6J6FC6Y7273HGJSCT7TTDFW55KCZFUOTAEBTWEJA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.888068836 (21m4.669002223s) [noise=true , address=104.36.82.15:4289 , piece_id=JHOTWB7GBMIQAGNLYGVGPNNSAF2CRSO7POJTDOQ56VSF5H55E37A, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.888347371 (21m4.669280756s) [noise=true , address=104.36.82.46:4513 , piece_id=VUGNBZAWSMCULWYJESRWAOWXGFHHXSI6FLMZVP44HFCW2EBBDR6A, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.888730452 (21m4.669663839s) [noise=true , address=104.36.82.22:4527 , piece_id=BE3D4HBJ7WVUAMEOHU6VMJ6EHFWO3IMGEXEJ5OJGRSFRSP7BOSJQ, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.889110640 (21m4.670044025s) [noise=true , address=79.127.205.233:4623 , piece_id=IMZL34MN2F4Z7YGCS3ITVX2D2CKPVPSS4PITRTK7VAAMJVER5SBA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.889204917 (21m4.670138299s) [noise=true , address=104.36.82.53:4122 , piece_id=3LJMB7ICJWLEX7XWELI322DBRINF7ADI67OOOAW6TWKLIGS5SBPA, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.889503339 (21m4.670436715s) [noise=true , address=104.36.82.20:4459 , piece_id=PFILN2FPLX4EIK5GTIXDBD5LIDOC5X5KA67ZTQ75OI4B2SCYPC5A, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.889801820 (21m4.670735196s) [noise=true , address=77.90.31.16:4766 , piece_id=QFSKGC4HJRXDS3PXP2OGNZDE7LJEM2CTOPPXSLWJO2FL3HO47CKQ, seg_pos={0 0}, upload=331]: Uploading piece... Jul 26 22:12:48.961563148 (21m4.742496535s) [upload=331]: Next returned nil segment Jul 26 22:12:48.961584888 (21m4.742518278s) [upload=331]: Waiting for error group managing segments... Jul 26 22:12:49.001854366 (21m4.782787751s) [noise=true , address=77.90.31.16:4766 , piece_id=QFSKGC4HJRXDS3PXP2OGNZDE7LJEM2CTOPPXSLWJO2FL3HO47CKQ, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:49.005189349 (21m4.786122718s) [noise=true , address=77.90.31.11:4579 , piece_id=MUZRLA67GHCABWG7NOTN2WILNUEAFHEO2JCZFURG6H4P3DXCE4IA, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:49.545610704 (21m5.326544076s) [noise=true , address=79.127.205.233:4623 , piece_id=IMZL34MN2F4Z7YGCS3ITVX2D2CKPVPSS4PITRTK7VAAMJVER5SBA, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:49.593626741 (21m5.374560106s) [noise=true , address=79.127.205.245:4231 , piece_id=G7OUM6FPKEKC6J6FC6Y7273HGJSCT7TTDFW55KCZFUOTAEBTWEJA, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:49.906649935 (21m5.687583311s) [noise=true , address=79.127.205.234:4612 , piece_id=OPG2RRVGEYTEI733WQIX5B2V7RETEGR6UH53DNAPIAUJED5GFJXA, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:50.411069913 (21m6.192003281s) [noise=true , address=79.127.201.215:4350 , piece_id=Y7EARFRFLNNZQMSCSL27OJ5P6DH42DJBZ5TWO7G7QUPKQKB5P2RQ, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:12:50.433510940 (21m6.21444434s) [noise=true , address=64.49.13.169:4716 , piece_id=2SBENU3JIVLK7JP77U2DNPZ5J2G5DR4TXMC5DQVPTC74MHXH7PEA, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:13:04.502088139 (21m20.283021504s) [noise=true , address=104.36.82.15:4289 , piece_id=JHOTWB7GBMIQAGNLYGVGPNNSAF2CRSO7POJTDOQ56VSF5H55E37A, seg_pos={0 0}, upload=331]: Done uploading piece. err: <nil> Jul 26 22:13:48.817133201 (22m4.598066577s) [noise=true , address=104.36.82.56:4251 , piece_id=JFJ33CH34YKRVXF2T3OWBX5NXU6NHCZMA7D7G6NITS4QR4A3OWTA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817195068 (22m4.598128434s) [noise=true , address=104.36.82.11:4133 , piece_id=JM2TNQTKXJD4LSUTALXU4A66MMNDDVLH6HC5XZFVTKLDY7ZX4ZUA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:12G5pngZWYQ2abKHEpM7Pa6YeCQ1LotX4qswbqHUwAy9H7a9Ym2, address:104.36.82.11:4133): protocol: rpc: dial tcp 104.36.82.11:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817251552 (22m4.598184926s) [noise=true , address=104.36.82.42:4809 , piece_id=WLJONQZYSEDY4OGJ5KCQPF7JUWJPRWZTG7NUAQY4JB7D3R3IGGTA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:1VZgucaa2gRuFhNxARMb7BDS6EGZXKChGPppbFksbPc5we7fTQ, address:104.36.82.42:4809): protocol: rpc: dial tcp 104.36.82.42:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817261989 (22m4.598195352s) [noise=true , address=104.36.82.22:4527 , piece_id=BE3D4HBJ7WVUAMEOHU6VMJ6EHFWO3IMGEXEJ5OJGRSFRSP7BOSJQ, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:1nzM7fbo5JececsEvKgB7yyYXyjUrveZ7dVfS7FawJvP8DfgZA, address:104.36.82.22:4527): protocol: write tcp 159.223.143.39:52596->104.36.82.22:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817300004 (22m4.598233379s) [noise=true , address=104.36.82.46:4513 , piece_id=VUGNBZAWSMCULWYJESRWAOWXGFHHXSI6FLMZVP44HFCW2EBBDR6A, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: rpc: dial tcp 104.36.82.46:4513: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817330388 (22m4.598263749s) [noise=true , address=104.36.82.53:4122 , piece_id=3LJMB7ICJWLEX7XWELI322DBRINF7ADI67OOOAW6TWKLIGS5SBPA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: write tcp 159.223.143.39:54884->104.36.82.53:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817345019 (22m4.598278385s) [noise=true , address=104.36.82.3:4783 , piece_id=MGNS6XRLLLKUEWGJLPM55KEV4F73LLTRXNH7S7ZKHKXIVCLPFGWA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:12ito72D7vikRdjK7BPt4LnhAK84bzi8AbZZPNWrTqmbjwQT1N, address:104.36.82.3:4783): protocol: write tcp 159.223.143.39:36370->104.36.82.3:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817388879 (22m4.598322241s) [noise=true , address=104.36.82.47:4536 , piece_id=QPKDGEP43676FQIFC47EKIP2TEEZUH2JF2Z6K4VR72MLC6COKSYA, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: write tcp 159.223.143.39:46980->104.36.82.47:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817400018 (22m4.598333383s) [noise=true , address=104.36.82.20:4459 , piece_id=PFILN2FPLX4EIK5GTIXDBD5LIDOC5X5KA67ZTQ75OI4B2SCYPC5A, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:1hpq5G467oUtEaE6Cj842pHCCZ1V7EraP5EDdGdWMxz4M3Q3pA, address:104.36.82.20:4459): protocol: write tcp 159.223.143.39:57166->104.36.82.20:4459: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817431070 (22m4.598364432s) [noise=true , address=104.36.82.45:4916 , piece_id=XOUNTRQKZFOHHP23CYGUVP7N5V4MHZLKKHVHMNQDLIVHELWPYL3A, seg_pos={0 0}, upload=331]: Done uploading piece. err: ecclient: upload failed (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa, address:104.36.82.45:4916): protocol: write tcp 159.223.143.39:40928->104.36.82.45:4916: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:48.817465252 (22m4.598398636s) [seg_pos={0 0}, upload=331]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:13:48.847184845 (22m4.628118231s) [upload=331]: Done uploading segments... Jul 26 22:13:48.847215243 (22m4.628148617s) [upload=331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.033301136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................XX..XXX..X..XX..X..XX.X....X.] duration: 393.679809ms FAIL (2025-07-26 22:13:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tocol: rpc: dial tcp 104.36.82.15:4257: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864074530 (20m15.64500791s) [noise=true , address=104.36.82.4:4802 , piece_id=VA26G7K6XQFH4PHZCJ2XWTGOD4GD5JR6DPVR35KCMLMWP4TGBXMA, seg_pos={0 0}, upload=320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:35088->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864116163 (20m15.645049543s) [seg_pos={0 0}, upload=320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:59.864361342 (20m15.645294726s) [upload=320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:59.864371913 (20m15.645305299s) [upload=320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:59.917691593 (20m15.698624962s) [upload=320]: Done uploading segments... Jul 26 22:11:59.917714191 (20m15.698647562s) [upload=320]: Upload finished. err: <nil> Jul 26 22:12:53.913781892 (21m9.69471528s) [upload=334]: Starting upload Jul 26 22:12:53.913910566 (21m9.694843946s) [upload=334]: Done starting upload Jul 26 22:12:53.914068111 (21m9.695001479s) [upload=334]: Uploading segments... Jul 26 22:12:53.914418778 (21m9.695352164s) [upload=334]: Got next segment. Inline: false Jul 26 22:12:53.914489104 (21m9.695422468s) [upload=334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:53.914499080 (21m9.695432464s) [upload=334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:53.964779415 (21m9.745712792s) [seg_pos={0 0}, upload=334]: Begin upload segment... Jul 26 22:12:53.964834832 (21m9.745768199s) [seg_pos={0 0}, upload=334]: Done begin upload segment. Jul 26 22:12:53.964861055 (21m9.745794419s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.964889604 (21m9.745822981s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965340371 (21m9.746273751s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965731230 (21m9.746664613s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966122432 (21m9.747055812s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966494352 (21m9.747427739s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966826509 (21m9.747759885s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967213838 (21m9.748147227s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967583090 (21m9.748516472s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969295495 (21m9.75022888s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969752444 (21m9.750685812s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970073562 (21m9.75100693s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970432919 (21m9.751366305s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.971368537 (21m9.752301923s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.972505536 (21m9.753438924s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.973901396 (21m9.754834793s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:54.096343157 (21m9.877276542s) [upload=334]: Next returned nil segment Jul 26 22:12:54.096365409 (21m9.877298804s) [upload=334]: Waiting for error group managing segments... Jul 26 22:12:54.102545693 (21m9.883479076s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103199713 (21m9.884133094s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103330731 (21m9.884264101s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103766692 (21m9.884700057s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104301838 (21m9.885235206s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104452646 (21m9.885386024s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104983988 (21m9.885917367s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105066863 (21m9.886000233s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105555466 (21m9.886488829s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105740907 (21m9.886674273s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108750596 (21m9.889683966s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108767895 (21m9.889701273s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:13:23.875895875 (21m39.656829258s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875916089 (21m39.656849455s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875996243 (21m39.656929626s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:33108->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876011671 (21m39.656945039s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:47806->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876035419 (21m39.656968803s) [seg_pos={0 0}, upload=334]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:13:23.902382513 (21m39.683315882s) [upload=334]: Done uploading segments... Jul 26 22:13:23.902433922 (21m39.683367306s) [upload=334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 393.70479ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................X..X........XXX.XXX.X.XX..X.] duration: 878.079195ms FAIL (2025-07-26 22:13:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 878.101508ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [................................XX..XXX..X..XX..X..XX.X....X] duration: 30.028923705s FAIL (2025-07-26 22:13:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tocol: rpc: dial tcp 104.36.82.15:4257: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864074530 (20m15.64500791s) [noise=true , address=104.36.82.4:4802 , piece_id=VA26G7K6XQFH4PHZCJ2XWTGOD4GD5JR6DPVR35KCMLMWP4TGBXMA, seg_pos={0 0}, upload=320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:35088->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864116163 (20m15.645049543s) [seg_pos={0 0}, upload=320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:59.864361342 (20m15.645294726s) [upload=320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:59.864371913 (20m15.645305299s) [upload=320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:59.917691593 (20m15.698624962s) [upload=320]: Done uploading segments... Jul 26 22:11:59.917714191 (20m15.698647562s) [upload=320]: Upload finished. err: <nil> Jul 26 22:12:53.913781892 (21m9.69471528s) [upload=334]: Starting upload Jul 26 22:12:53.913910566 (21m9.694843946s) [upload=334]: Done starting upload Jul 26 22:12:53.914068111 (21m9.695001479s) [upload=334]: Uploading segments... Jul 26 22:12:53.914418778 (21m9.695352164s) [upload=334]: Got next segment. Inline: false Jul 26 22:12:53.914489104 (21m9.695422468s) [upload=334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:53.914499080 (21m9.695432464s) [upload=334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:53.964779415 (21m9.745712792s) [seg_pos={0 0}, upload=334]: Begin upload segment... Jul 26 22:12:53.964834832 (21m9.745768199s) [seg_pos={0 0}, upload=334]: Done begin upload segment. Jul 26 22:12:53.964861055 (21m9.745794419s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.964889604 (21m9.745822981s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965340371 (21m9.746273751s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965731230 (21m9.746664613s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966122432 (21m9.747055812s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966494352 (21m9.747427739s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966826509 (21m9.747759885s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967213838 (21m9.748147227s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967583090 (21m9.748516472s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969295495 (21m9.75022888s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969752444 (21m9.750685812s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970073562 (21m9.75100693s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970432919 (21m9.751366305s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.971368537 (21m9.752301923s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.972505536 (21m9.753438924s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.973901396 (21m9.754834793s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:54.096343157 (21m9.877276542s) [upload=334]: Next returned nil segment Jul 26 22:12:54.096365409 (21m9.877298804s) [upload=334]: Waiting for error group managing segments... Jul 26 22:12:54.102545693 (21m9.883479076s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103199713 (21m9.884133094s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103330731 (21m9.884264101s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103766692 (21m9.884700057s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104301838 (21m9.885235206s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104452646 (21m9.885386024s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104983988 (21m9.885917367s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105066863 (21m9.886000233s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105555466 (21m9.886488829s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105740907 (21m9.886674273s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108750596 (21m9.889683966s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108767895 (21m9.889701273s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:13:23.875895875 (21m39.656829258s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875916089 (21m39.656849455s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875996243 (21m39.656929626s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:33108->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876011671 (21m39.656945039s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:47806->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876035419 (21m39.656968803s) [seg_pos={0 0}, upload=334]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:13:23.902382513 (21m39.683315882s) [upload=334]: Done uploading segments... Jul 26 22:13:23.902433922 (21m39.683367306s) [upload=334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.028958492s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................................X..X........XXX.XXX.X.XX..X] duration: 30.029171989s FAIL (2025-07-26 22:13:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.02923937s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 1m0.03666735s FAIL (2025-07-26 22:12:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== FVRTGARO4O32ANA43KXQRURW3CT4JHM23EABKQ, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.771687722 (20m4.552621115s) [noise=true , address=104.36.82.38:4720 , piece_id=CUPQIGQJINBN3FUBBRNUMPND2A2RBSLXR5B6ZK2L3CMXQXW7BL2A, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.772048864 (20m4.552982248s) [noise=true , address=104.36.82.40:4958 , piece_id=X5FTVSFKOVM7I4XCSSBIPNMB4ITI7I5A2QNRIZQUMSWC7YHD7BPQ, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.772109465 (20m4.553042851s) [noise=true , address=104.36.82.36:4622 , piece_id=JMRSHJZDCSX7EXJ5KWCKQ6JH4FEZQOE4ISWWYNI3AFIRWGRHU3WA, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.772578398 (20m4.553511765s) [noise=true , address=104.36.82.46:4507 , piece_id=YLU5K5VJ7ZYSXIG7HWCJEENBSGTIKFP76523SU6PVEA4R3ZNUXHQ, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.772645227 (20m4.553578608s) [noise=true , address=104.36.82.45:4914 , piece_id=THXVJBZMHLSGHCZG4ZNACHYXV3L75BVMCJ424DMERWKRYE2KTHTA, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.772988214 (20m4.553921599s) [noise=true , address=104.36.82.53:4148 , piece_id=VEXQSV6IAS6TLWNCF3PUJUQDUT2WK73M25XZZXLHCSD5RRKVJMHA, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.773066422 (20m4.553999803s) [noise=true , address=79.127.219.38:4009 , piece_id=6JRRXSTQBE6EOFDUXD5KEH6RN432JMJ2OYYLBMYEZA6KE243JY2Q, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.773455080 (20m4.554388461s) [noise=true , address=104.36.82.54:4179 , piece_id=RGMENJDYMHWSHMEYVADSRMOLBUB6IT33325VW3JPO7XK56ZITDPQ, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.773875430 (20m4.554808811s) [noise=true , address=45.91.116.14:4222 , piece_id=CM4WGAHYPL4QN2Q6YFV574DLCKGIVQCZJLJEZGB65U3QAR72ZBWA, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.774253585 (20m4.555186966s) [noise=true , address=79.127.201.215:4349 , piece_id=KVUNSC6GMFECINGBAWQTV3Y3UYWGQ2J54DD4OCEFC6MCHOVARDEA, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.774577364 (20m4.555510748s) [noise=true , address=104.36.82.15:4262 , piece_id=I4ZCPEPWDXVMFWYUFO2Z4GXAMKOSVSNRCSGLLGRANDFPX7TFSTSQ, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.774878014 (20m4.555811392s) [noise=true , address=104.36.82.56:4246 , piece_id=RUMT7IC6MESYCSHV3CGWWHYGID3VD6MVDGFBUQY2OZLAD256GRHA, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.775185857 (20m4.556119242s) [noise=true , address=104.36.82.52:4105 , piece_id=TTV5ITJBS27FVEGUYQOBAV3NBNOJ5INFTT33YCWGG2ESOUIARMBQ, seg_pos={0 0}, upload=315]: Uploading piece... Jul 26 22:11:48.843351142 (20m4.624284518s) [upload=315]: Next returned nil segment Jul 26 22:11:48.843368301 (20m4.62430167s) [upload=315]: Waiting for error group managing segments... Jul 26 22:11:49.401370215 (20m5.182303576s) [noise=true , address=79.127.205.233:4645 , piece_id=WBVARFUWLBUQGTZ56TF4CDYTYEXQTDLT7YTLMYCQPB7375P4L34A, seg_pos={0 0}, upload=315]: Done uploading piece. err: <nil> Jul 26 22:11:49.453616978 (20m5.234550344s) [noise=true , address=79.127.205.245:4251 , piece_id=MRX2VWPOB2DACEC5RYBCCG32GHFPPBAQOMC4X42UDDJNPZL7X2PA, seg_pos={0 0}, upload=315]: Done uploading piece. err: <nil> Jul 26 22:11:50.054877017 (20m5.83581039s) [noise=true , address=79.127.219.38:4009 , piece_id=6JRRXSTQBE6EOFDUXD5KEH6RN432JMJ2OYYLBMYEZA6KE243JY2Q, seg_pos={0 0}, upload=315]: Done uploading piece. err: <nil> Jul 26 22:11:51.661179744 (20m7.44211312s) [noise=true , address=45.91.116.14:4222 , piece_id=CM4WGAHYPL4QN2Q6YFV574DLCKGIVQCZJLJEZGB65U3QAR72ZBWA, seg_pos={0 0}, upload=315]: Done uploading piece. err: <nil> Jul 26 22:11:57.160340813 (20m12.941274195s) [noise=true , address=79.127.201.215:4349 , piece_id=KVUNSC6GMFECINGBAWQTV3Y3UYWGQ2J54DD4OCEFC6MCHOVARDEA, seg_pos={0 0}, upload=315]: Done uploading piece. err: <nil> Jul 26 22:12:48.651609751 (21m4.432543128s) [noise=true , address=104.36.82.8:4974 , piece_id=DIVQV4DMZ5ZA7OFVRTGARO4O32ANA43KXQRURW3CT4JHM23EABKQ, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1, address:104.36.82.8:4974): protocol: rpc: dial tcp 104.36.82.8:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.651686514 (21m4.432619891s) [noise=true , address=104.36.82.40:4958 , piece_id=X5FTVSFKOVM7I4XCSSBIPNMB4ITI7I5A2QNRIZQUMSWC7YHD7BPQ, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:1ySYdoPKsX8RRcnfGuneBqVjXLbuDQJ7ofRiv6dEyYxR32hoGR, address:104.36.82.40:4958): protocol: rpc: dial tcp 104.36.82.40:4958: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.651738915 (21m4.432672291s) [noise=true , address=104.36.82.47:4535 , piece_id=YB6ZIYVJOPGAI77OLGFDWLBYHUMFH2DDNJKL5XG265VAUACSOJNA, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.651793226 (21m4.432726602s) [noise=true , address=104.36.82.55:4202 , piece_id=XH6PRPANRALQD2RBOZZNNHJ3TEDWPJSI6XHXNP5YLXIWPNPUILIQ, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.651866562 (21m4.432799941s) [noise=true , address=104.36.82.45:4914 , piece_id=THXVJBZMHLSGHCZG4ZNACHYXV3L75BVMCJ424DMERWKRYE2KTHTA, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:12wWwD62SvgY72yFsUXtZMMvCea2xZkRtq72sB7aAkjHoyJWa6N, address:104.36.82.45:4914): protocol: rpc: dial tcp 104.36.82.45:4914: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.651880902 (21m4.432814286s) [noise=true , address=104.36.82.46:4507 , piece_id=YLU5K5VJ7ZYSXIG7HWCJEENBSGTIKFP76523SU6PVEA4R3ZNUXHQ, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: rpc: dial tcp 104.36.82.46:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.651926302 (21m4.432859681s) [noise=true , address=104.36.82.53:4148 , piece_id=VEXQSV6IAS6TLWNCF3PUJUQDUT2WK73M25XZZXLHCSD5RRKVJMHA, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: rpc: dial tcp 104.36.82.53:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.651987658 (21m4.432921035s) [noise=true , address=104.36.82.56:4246 , piece_id=RUMT7IC6MESYCSHV3CGWWHYGID3VD6MVDGFBUQY2OZLAD256GRHA, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:1XtKegw9zcPDGZu9z7STKgAzwx5KFrkB9qCgdNUe6XqZ6tzzj9, address:104.36.82.56:4246): protocol: rpc: dial tcp 104.36.82.56:4246: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.652000046 (21m4.432933429s) [noise=true , address=104.36.82.54:4179 , piece_id=RGMENJDYMHWSHMEYVADSRMOLBUB6IT33325VW3JPO7XK56ZITDPQ, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP, address:104.36.82.54:4179): protocol: rpc: dial tcp 104.36.82.54:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.652046196 (21m4.432979577s) [noise=true , address=104.36.82.38:4720 , piece_id=CUPQIGQJINBN3FUBBRNUMPND2A2RBSLXR5B6ZK2L3CMXQXW7BL2A, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:12Y16Z5H7jpsCyRzsAajF5AR3mURqTY8a4jgEr2xP4Yxy9W9kdC, address:104.36.82.38:4720): protocol: write tcp 159.223.143.39:49654->104.36.82.38:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.652094083 (21m4.433027463s) [noise=true , address=104.36.82.52:4105 , piece_id=TTV5ITJBS27FVEGUYQOBAV3NBNOJ5INFTT33YCWGG2ESOUIARMBQ, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:104.36.82.52:4105): protocol: rpc: dial tcp 104.36.82.52:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.652242616 (21m4.433175997s) [noise=true , address=104.36.82.36:4622 , piece_id=JMRSHJZDCSX7EXJ5KWCKQ6JH4FEZQOE4ISWWYNI3AFIRWGRHU3WA, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:1xEK5XL3sYcip7PsQcZBQYihgnEZykdMmsKkCr2TUmbTFYncfs, address:104.36.82.36:4622): protocol: rpc: dial tcp 104.36.82.36:4622: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.652292032 (21m4.433225416s) [noise=true , address=104.36.82.15:4262 , piece_id=I4ZCPEPWDXVMFWYUFO2Z4GXAMKOSVSNRCSGLLGRANDFPX7TFSTSQ, seg_pos={0 0}, upload=315]: Done uploading piece. err: ecclient: upload failed (node:12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, address:104.36.82.15:4262): protocol: rpc: dial tcp 104.36.82.15:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:48.652327685 (21m4.433261072s) [seg_pos={0 0}, upload=315]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:12:48.684671380 (21m4.465604759s) [upload=315]: Done uploading segments... Jul 26 22:12:48.684719152 (21m4.465652529s) [upload=315]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.036691155s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 482.713067ms FAIL (2025-07-26 22:12:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.747908ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................................XX..XXX..X..XX..X..XX.X....] duration: 783.209029ms FAIL (2025-07-26 22:12:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:22.534459505 (16m38.315392869s) [noise=true , address=104.36.82.3:4765 , piece_id=QY5IWCE4YCDVGCA54ZNJR5CNH5POOREICCZCM6SNGI3I3QPZPTWQ, seg_pos={0 0}, upload=263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: write tcp 159.223.143.39:34250->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:22.534491171 (16m38.315424558s) [seg_pos={0 0}, upload=263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:08:22.534577358 (16m38.315510743s) [upload=263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:08:22.534582422 (16m38.315515801s) [upload=263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:08:22.575438889 (16m38.356372275s) [upload=263]: Done uploading segments... Jul 26 22:08:22.575462290 (16m38.356395666s) [upload=263]: Upload finished. err: <nil> Jul 26 22:09:12.073816930 (17m27.854750309s) [upload=276]: Starting upload Jul 26 22:09:12.073908198 (17m27.854841586s) [upload=276]: Done starting upload Jul 26 22:09:12.074014224 (17m27.854947604s) [upload=276]: Uploading segments... Jul 26 22:09:12.074306818 (17m27.8552402s) [upload=276]: Got next segment. Inline: false Jul 26 22:09:12.074373049 (17m27.855306427s) [upload=276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:09:12.074386765 (17m27.855320186s) [upload=276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:09:12.102208192 (17m27.883141579s) [seg_pos={0 0}, upload=276]: Begin upload segment... Jul 26 22:09:12.102281970 (17m27.883215358s) [seg_pos={0 0}, upload=276]: Done begin upload segment. Jul 26 22:09:12.102316815 (17m27.883250203s) [noise=true , address=104.36.82.6:4873 , piece_id=PC4W673ITXSO7A6OR5XR6J6K562JMQHJUNEDLWPWH3JJOLDDXALQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.103398160 (17m27.884331551s) [noise=true , address=45.91.116.5:4297 , piece_id=XCNIJJWIT3OMAK2GQJB7KBGAJHPHTL7GVWCCLCKDH2MK4DXWT6DQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.103834590 (17m27.884767971s) [noise=true , address=104.36.82.8:4970 , piece_id=I6WEZKRVPH3HR4HHMETUYYJSMHWRWXN4MU5YAZHTY5KPGIFY3GKA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.104321474 (17m27.885254862s) [noise=true , address=77.90.31.18:4807 , piece_id=KC253KPUC73VN7SPMTJZUX2P6WJ6BK4ODCYBRG6HISIFDBCSJL6A, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.105886362 (17m27.886819751s) [noise=true , address=45.91.116.4:4939 , piece_id=CHGWUUTASHFZYXQOXXWSEXRKNI7FVTRALPKD4TVZM5CGBRWNOWHA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.106338669 (17m27.887272047s) [noise=true , address=77.90.31.14:4663 , piece_id=4RJ57NHGYGNXNZ6WKJFSYTXABP5TCUW6SJ7N7BK6JYUI3CVXE6HQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.106682458 (17m27.887615847s) [noise=true , address=104.36.82.15:4258 , piece_id=GWV7CGQUBC6KBHB5CPSUAKGEXZIRV554OMJLZCMV2FNCYDRPIHIQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.107171678 (17m27.888105069s) [noise=true , address=77.90.31.16:4734 , piece_id=KBQPSDTMMERJGT3L4L7WH536IO7LESN3RBNGGHVGG7UM5ABQKBQQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.107598895 (17m27.888532281s) [noise=true , address=45.91.116.5:4299 , piece_id=MA22QRMEKYQGC4JHC73VHRZYUON6X5GSYCKPALUILHOTVE2JALZQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.110879103 (17m27.891812491s) [noise=true , address=45.91.116.10:4083 , piece_id=ULUVLWJPQ4GZASQJEH656LNH3LTDG437GOWBW2GCCVYALVSCLP2Q, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111179084 (17m27.892112471s) [noise=true , address=45.91.116.14:4190 , piece_id=YEFEQKDNF3RFW4YEUJG4CDDX2T4W5PYG2EBXZ6HOJTTGVD2L6LLQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111561235 (17m27.892494622s) [noise=true , address=45.91.116.7:4011 , piece_id=2Q3VP4NNXV7FSIB3KTDKEI5VEA6FMJM3DV7WVXARUAAXIMVYPP5Q, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111944292 (17m27.892877681s) [noise=true , address=104.36.82.7:4911 , piece_id=GFYIHUPEHJCF75CAFM4CZ7KUK4XD445KOF7UBVDM4HXMHRPWZYZQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.112357623 (17m27.893291011s) [noise=true , address=45.91.116.8:4047 , piece_id=M6TIGG6K5IY5YLCWIIOAYM4OADHYT654KAHFZLAVHYRZFQXOMNSA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.112763233 (17m27.893696631s) [noise=true , address=45.91.116.9:4262 , piece_id=7ROQYHMFIZSJE3E5IBV2YYRFBOKXZFPP4APP64TLO7EO2XTQFN3A, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.113189270 (17m27.894122659s) [noise=true , address=45.91.116.7:4010 , piece_id=4TGUR3523QK6DZ7ZNPLQIGMN5L6TM5XXQFDTIPUBVUSNMUALMTXQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.237315416 (17m28.018248798s) [upload=276]: Next returned nil segment Jul 26 22:09:12.237338503 (17m28.018271879s) [upload=276]: Waiting for error group managing segments... Jul 26 22:09:12.242491332 (17m28.023424712s) [noise=true , address=77.90.31.18:4807 , piece_id=KC253KPUC73VN7SPMTJZUX2P6WJ6BK4ODCYBRG6HISIFDBCSJL6A, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.242526026 (17m28.023459402s) [noise=true , address=45.91.116.4:4939 , piece_id=CHGWUUTASHFZYXQOXXWSEXRKNI7FVTRALPKD4TVZM5CGBRWNOWHA, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.244975691 (17m28.025909061s) [noise=true , address=45.91.116.5:4297 , piece_id=XCNIJJWIT3OMAK2GQJB7KBGAJHPHTL7GVWCCLCKDH2MK4DXWT6DQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.245067069 (17m28.026000444s) [noise=true , address=45.91.116.7:4011 , piece_id=2Q3VP4NNXV7FSIB3KTDKEI5VEA6FMJM3DV7WVXARUAAXIMVYPP5Q, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.245749945 (17m28.026683314s) [noise=true , address=45.91.116.8:4047 , piece_id=M6TIGG6K5IY5YLCWIIOAYM4OADHYT654KAHFZLAVHYRZFQXOMNSA, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.246361942 (17m28.02729532s) [noise=true , address=45.91.116.5:4299 , piece_id=MA22QRMEKYQGC4JHC73VHRZYUON6X5GSYCKPALUILHOTVE2JALZQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247224019 (17m28.028157391s) [noise=true , address=77.90.31.14:4663 , piece_id=4RJ57NHGYGNXNZ6WKJFSYTXABP5TCUW6SJ7N7BK6JYUI3CVXE6HQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247647098 (17m28.028580476s) [noise=true , address=45.91.116.10:4083 , piece_id=ULUVLWJPQ4GZASQJEH656LNH3LTDG437GOWBW2GCCVYALVSCLP2Q, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247771695 (17m28.028705064s) [noise=true , address=45.91.116.14:4190 , piece_id=YEFEQKDNF3RFW4YEUJG4CDDX2T4W5PYG2EBXZ6HOJTTGVD2L6LLQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247883972 (17m28.028817342s) [noise=true , address=45.91.116.9:4262 , piece_id=7ROQYHMFIZSJE3E5IBV2YYRFBOKXZFPP4APP64TLO7EO2XTQFN3A, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.248440176 (17m28.029373547s) [noise=true , address=45.91.116.7:4010 , piece_id=4TGUR3523QK6DZ7ZNPLQIGMN5L6TM5XXQFDTIPUBVUSNMUALMTXQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.291635996 (17m28.07256936s) [noise=true , address=77.90.31.16:4734 , piece_id=KBQPSDTMMERJGT3L4L7WH536IO7LESN3RBNGGHVGG7UM5ABQKBQQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:42.031234530 (17m57.812167916s) [noise=true , address=104.36.82.8:4970 , piece_id=I6WEZKRVPH3HR4HHMETUYYJSMHWRWXN4MU5YAZHTY5KPGIFY3GKA, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031315006 (17m57.812248388s) [noise=true , address=104.36.82.15:4258 , piece_id=GWV7CGQUBC6KBHB5CPSUAKGEXZIRV554OMJLZCMV2FNCYDRPIHIQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:41696->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031326322 (17m57.812259705s) [noise=true , address=104.36.82.7:4911 , piece_id=GFYIHUPEHJCF75CAFM4CZ7KUK4XD445KOF7UBVDM4HXMHRPWZYZQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:42482->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031335779 (17m57.812269142s) [noise=true , address=104.36.82.6:4873 , piece_id=PC4W673ITXSO7A6OR5XR6J6K562JMQHJUNEDLWPWH3JJOLDDXALQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031372559 (17m57.812305942s) [seg_pos={0 0}, upload=276]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:09:42.054859127 (17m57.835792503s) [upload=276]: Done uploading segments... Jul 26 22:09:42.054915101 (17m57.835848484s) [upload=276]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 783.268058ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [..................................X..X........XXX.XXX.X.XX..] duration: 1.826433079s FAIL (2025-07-26 22:12:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.826463188s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.973851193s FAIL (2025-07-26 22:11:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.973879652s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 1m0.029425173s FAIL (2025-07-26 22:11:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 452520767s) [noise=true , address=104.36.82.52:4107 , piece_id=U2PF6QJ5OOPU7EOJLULKL5UBMBZYC54MHLI7P3UK3RLR7FGXABFA, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.671869758 (19m4.452803136s) [noise=true , address=104.36.82.40:4950 , piece_id=BBJJNRAS5BWRNFMPAFPKHSA6V2SFBJ5SYXP4WXZZGM65B4XKL2EA, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.671983793 (19m4.452917176s) [noise=true , address=104.36.82.44:4885 , piece_id=53MNN6UPE4UTMM5B7S76SL4ZZ3HEEK33CRWQ4IEJOTHWDQYFE2RA, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.672520060 (19m4.453453445s) [noise=true , address=104.36.82.39:4750 , piece_id=3FETCM4BQGEQCKNOR26V4BWKYXZISNW6APEX6KSAYQM34QFZOLAQ, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.672952369 (19m4.453885754s) [noise=true , address=79.127.201.215:4353 , piece_id=XD4NFAFGIQ5B7FFZNZOBJSEI43NE5ABCHIPZPR4IDEBHRYE5A3VQ, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.673355762 (19m4.45428915s) [noise=true , address=104.36.82.4:4823 , piece_id=73CTTJGCSB2BYLXFMZ7UPH2LP2E73557WBPQYHYIPNR2D3DUV54Q, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.673764391 (19m4.454697979s) [noise=true , address=45.91.116.14:4206 , piece_id=2OY7SMVUHPVIMFU7CGN7VPXSYODYFQ567RF72GBOEM5TWEOLQWJQ, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.674145080 (19m4.455078465s) [noise=true , address=79.127.205.234:4604 , piece_id=3KY6HIFOJQ6XZWERSI76VGMRHTXML5SONBGT7PTWT6IT7BQFKO6A, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.674510466 (19m4.455443851s) [noise=true , address=79.127.205.245:4248 , piece_id=SHU7GECJ5HLLBHIMFPBEU5AD3IRRI22A3LDG5MM23OT3FGCHW6KA, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.674866605 (19m4.455799992s) [noise=true , address=104.36.82.53:4127 , piece_id=5XFSYIWVOMOLRIQ4PEEO7Q6SWSHT7EQX3HL2533PXNWDVVE4S6UQ, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.675339637 (19m4.456273028s) [noise=true , address=104.36.82.44:4900 , piece_id=34PYCP5PW4Y4TCGACKPT5AVGMUAHXZVVJITQUWXQNOEGPHRQASJA, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.675746907 (19m4.456680294s) [noise=true , address=104.36.82.47:4525 , piece_id=GECSJ3MZNMV2VZWKXNR6SEEBQVEDJ3UBLRTIUYMZRSWOBDNY7NWQ, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.676100619 (19m4.457034001s) [noise=true , address=79.127.219.48:4075 , piece_id=UGDNM4AXLKTZST7APDV6OZIG3HYBXPFOU5CJKBCB2VJERHQ364XA, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.677307436 (19m4.458240822s) [noise=true , address=104.36.82.13:4194 , piece_id=UJTHSK7SMT635X62AAHCWBLYN7PEYUI5ENFEJTUJJ3DRFZPEDZJQ, seg_pos={0 0}, upload=299]: Uploading piece... Jul 26 22:10:48.754503774 (19m4.535437162s) [upload=299]: Next returned nil segment Jul 26 22:10:48.754525287 (19m4.535458675s) [upload=299]: Waiting for error group managing segments... Jul 26 22:10:49.257392427 (19m5.038325808s) [noise=true , address=79.127.205.234:4604 , piece_id=3KY6HIFOJQ6XZWERSI76VGMRHTXML5SONBGT7PTWT6IT7BQFKO6A, seg_pos={0 0}, upload=299]: Done uploading piece. err: <nil> Jul 26 22:10:49.390070743 (19m5.171004112s) [noise=true , address=79.127.205.245:4248 , piece_id=SHU7GECJ5HLLBHIMFPBEU5AD3IRRI22A3LDG5MM23OT3FGCHW6KA, seg_pos={0 0}, upload=299]: Done uploading piece. err: <nil> Jul 26 22:10:51.342782311 (19m7.123715677s) [noise=true , address=79.127.201.215:4353 , piece_id=XD4NFAFGIQ5B7FFZNZOBJSEI43NE5ABCHIPZPR4IDEBHRYE5A3VQ, seg_pos={0 0}, upload=299]: Done uploading piece. err: <nil> Jul 26 22:10:55.051630945 (19m10.832564317s) [noise=true , address=79.127.219.48:4075 , piece_id=UGDNM4AXLKTZST7APDV6OZIG3HYBXPFOU5CJKBCB2VJERHQ364XA, seg_pos={0 0}, upload=299]: Done uploading piece. err: <nil> Jul 26 22:10:56.507105697 (19m12.288039074s) [noise=true , address=45.91.116.14:4206 , piece_id=2OY7SMVUHPVIMFU7CGN7VPXSYODYFQ567RF72GBOEM5TWEOLQWJQ, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:10:59.318383667 (19m15.099317272s) [noise=true , address=45.91.116.14:4222 , piece_id=4DH7SUPABSXNCDNRUQZ2P3XLROV4KUMLOA57B3F3ZMI5JPS7VX6Q, seg_pos={0 0}, upload=299]: Done uploading piece. err: <nil> Jul 26 22:11:48.538345225 (20m4.319278632s) [noise=true , address=104.36.82.44:4900 , piece_id=34PYCP5PW4Y4TCGACKPT5AVGMUAHXZVVJITQUWXQNOEGPHRQASJA, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:1R8K65zsCspu7W2oAnvtpz7zEsQFkDG1e5BAp6faQYMQrkh4oQ, address:104.36.82.44:4900): protocol: rpc: dial tcp 104.36.82.44:4900: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.538503588 (20m4.319436968s) [noise=true , address=104.36.82.54:4190 , piece_id=NJLFKOTK6VCMZJOATUA55CCW27ZXID5JW6FKJHF5TQWOV6NMP4ZA, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:11:48.538593727 (20m4.319527109s) [noise=true , address=104.36.82.52:4107 , piece_id=U2PF6QJ5OOPU7EOJLULKL5UBMBZYC54MHLI7P3UK3RLR7FGXABFA, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: rpc: dial tcp 104.36.82.52:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.538678333 (20m4.319611699s) [noise=true , address=104.36.82.53:4127 , piece_id=5XFSYIWVOMOLRIQ4PEEO7Q6SWSHT7EQX3HL2533PXNWDVVE4S6UQ, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:12VM12t6Dq6SyApviDYSSUqWnnzW3wTfA4o8VsvJaah4fLj9EWh, address:104.36.82.53:4127): protocol: rpc: dial tcp 104.36.82.53:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.539384632 (20m4.320318018s) [noise=true , address=104.36.82.44:4885 , piece_id=53MNN6UPE4UTMM5B7S76SL4ZZ3HEEK33CRWQ4IEJOTHWDQYFE2RA, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: rpc: dial tcp 104.36.82.44:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.539483023 (20m4.320416412s) [noise=true , address=104.36.82.47:4525 , piece_id=GECSJ3MZNMV2VZWKXNR6SEEBQVEDJ3UBLRTIUYMZRSWOBDNY7NWQ, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: rpc: dial tcp 104.36.82.47:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.539512272 (20m4.320445654s) [noise=true , address=104.36.82.13:4194 , piece_id=UJTHSK7SMT635X62AAHCWBLYN7PEYUI5ENFEJTUJJ3DRFZPEDZJQ, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:1u6tGwzk56SwgUqzpyRFLNTTysUtk6C2Ls77UkjFryeuF9D8pV, address:104.36.82.13:4194): protocol: rpc: dial tcp 104.36.82.13:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.539592375 (20m4.32052576s) [noise=true , address=104.36.82.4:4823 , piece_id=73CTTJGCSB2BYLXFMZ7UPH2LP2E73557WBPQYHYIPNR2D3DUV54Q, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:1JWhB5QxbZsTD863tEfkEN1CUbKBMQvMJ9ZaiMAwKvxxXyDkEq, address:104.36.82.4:4823): protocol: rpc: dial tcp 104.36.82.4:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.539652248 (20m4.320585628s) [noise=true , address=104.36.82.46:4490 , piece_id=G5GJIVXRCKB442EDM6VXH3LSPYDZVMYOHBKYPOLG3WSNKWPPCDUA, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: rpc: dial tcp 104.36.82.46:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.539682054 (20m4.320615439s) [noise=true , address=104.36.82.40:4950 , piece_id=BBJJNRAS5BWRNFMPAFPKHSA6V2SFBJ5SYXP4WXZZGM65B4XKL2EA, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk, address:104.36.82.40:4950): protocol: write tcp 159.223.143.39:47272->104.36.82.40:4950: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.539696646 (20m4.320630019s) [noise=true , address=104.36.82.42:4830 , piece_id=AQNZXTYFGMZFN5MRH77SNY2I7SIQF7Y3MER5VW4I52APZDN5A2CA, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.539723483 (20m4.320656869s) [noise=true , address=104.36.82.39:4750 , piece_id=3FETCM4BQGEQCKNOR26V4BWKYXZISNW6APEX6KSAYQM34QFZOLAQ, seg_pos={0 0}, upload=299]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:104.36.82.39:4750): protocol: write tcp 159.223.143.39:48590->104.36.82.39:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:48.539788261 (20m4.320721657s) [seg_pos={0 0}, upload=299]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:11:48.563741213 (20m4.344674596s) [upload=299]: Done uploading segments... Jul 26 22:11:48.563784754 (20m4.344718136s) [upload=299]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.029466925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..................................XX..XXX..X..XX..X..XX.X...] duration: 802.03034ms FAIL (2025-07-26 22:11:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:22.534459505 (16m38.315392869s) [noise=true , address=104.36.82.3:4765 , piece_id=QY5IWCE4YCDVGCA54ZNJR5CNH5POOREICCZCM6SNGI3I3QPZPTWQ, seg_pos={0 0}, upload=263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: write tcp 159.223.143.39:34250->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:22.534491171 (16m38.315424558s) [seg_pos={0 0}, upload=263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:08:22.534577358 (16m38.315510743s) [upload=263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:08:22.534582422 (16m38.315515801s) [upload=263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:08:22.575438889 (16m38.356372275s) [upload=263]: Done uploading segments... Jul 26 22:08:22.575462290 (16m38.356395666s) [upload=263]: Upload finished. err: <nil> Jul 26 22:09:12.073816930 (17m27.854750309s) [upload=276]: Starting upload Jul 26 22:09:12.073908198 (17m27.854841586s) [upload=276]: Done starting upload Jul 26 22:09:12.074014224 (17m27.854947604s) [upload=276]: Uploading segments... Jul 26 22:09:12.074306818 (17m27.8552402s) [upload=276]: Got next segment. Inline: false Jul 26 22:09:12.074373049 (17m27.855306427s) [upload=276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:09:12.074386765 (17m27.855320186s) [upload=276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:09:12.102208192 (17m27.883141579s) [seg_pos={0 0}, upload=276]: Begin upload segment... Jul 26 22:09:12.102281970 (17m27.883215358s) [seg_pos={0 0}, upload=276]: Done begin upload segment. Jul 26 22:09:12.102316815 (17m27.883250203s) [noise=true , address=104.36.82.6:4873 , piece_id=PC4W673ITXSO7A6OR5XR6J6K562JMQHJUNEDLWPWH3JJOLDDXALQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.103398160 (17m27.884331551s) [noise=true , address=45.91.116.5:4297 , piece_id=XCNIJJWIT3OMAK2GQJB7KBGAJHPHTL7GVWCCLCKDH2MK4DXWT6DQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.103834590 (17m27.884767971s) [noise=true , address=104.36.82.8:4970 , piece_id=I6WEZKRVPH3HR4HHMETUYYJSMHWRWXN4MU5YAZHTY5KPGIFY3GKA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.104321474 (17m27.885254862s) [noise=true , address=77.90.31.18:4807 , piece_id=KC253KPUC73VN7SPMTJZUX2P6WJ6BK4ODCYBRG6HISIFDBCSJL6A, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.105886362 (17m27.886819751s) [noise=true , address=45.91.116.4:4939 , piece_id=CHGWUUTASHFZYXQOXXWSEXRKNI7FVTRALPKD4TVZM5CGBRWNOWHA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.106338669 (17m27.887272047s) [noise=true , address=77.90.31.14:4663 , piece_id=4RJ57NHGYGNXNZ6WKJFSYTXABP5TCUW6SJ7N7BK6JYUI3CVXE6HQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.106682458 (17m27.887615847s) [noise=true , address=104.36.82.15:4258 , piece_id=GWV7CGQUBC6KBHB5CPSUAKGEXZIRV554OMJLZCMV2FNCYDRPIHIQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.107171678 (17m27.888105069s) [noise=true , address=77.90.31.16:4734 , piece_id=KBQPSDTMMERJGT3L4L7WH536IO7LESN3RBNGGHVGG7UM5ABQKBQQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.107598895 (17m27.888532281s) [noise=true , address=45.91.116.5:4299 , piece_id=MA22QRMEKYQGC4JHC73VHRZYUON6X5GSYCKPALUILHOTVE2JALZQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.110879103 (17m27.891812491s) [noise=true , address=45.91.116.10:4083 , piece_id=ULUVLWJPQ4GZASQJEH656LNH3LTDG437GOWBW2GCCVYALVSCLP2Q, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111179084 (17m27.892112471s) [noise=true , address=45.91.116.14:4190 , piece_id=YEFEQKDNF3RFW4YEUJG4CDDX2T4W5PYG2EBXZ6HOJTTGVD2L6LLQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111561235 (17m27.892494622s) [noise=true , address=45.91.116.7:4011 , piece_id=2Q3VP4NNXV7FSIB3KTDKEI5VEA6FMJM3DV7WVXARUAAXIMVYPP5Q, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111944292 (17m27.892877681s) [noise=true , address=104.36.82.7:4911 , piece_id=GFYIHUPEHJCF75CAFM4CZ7KUK4XD445KOF7UBVDM4HXMHRPWZYZQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.112357623 (17m27.893291011s) [noise=true , address=45.91.116.8:4047 , piece_id=M6TIGG6K5IY5YLCWIIOAYM4OADHYT654KAHFZLAVHYRZFQXOMNSA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.112763233 (17m27.893696631s) [noise=true , address=45.91.116.9:4262 , piece_id=7ROQYHMFIZSJE3E5IBV2YYRFBOKXZFPP4APP64TLO7EO2XTQFN3A, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.113189270 (17m27.894122659s) [noise=true , address=45.91.116.7:4010 , piece_id=4TGUR3523QK6DZ7ZNPLQIGMN5L6TM5XXQFDTIPUBVUSNMUALMTXQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.237315416 (17m28.018248798s) [upload=276]: Next returned nil segment Jul 26 22:09:12.237338503 (17m28.018271879s) [upload=276]: Waiting for error group managing segments... Jul 26 22:09:12.242491332 (17m28.023424712s) [noise=true , address=77.90.31.18:4807 , piece_id=KC253KPUC73VN7SPMTJZUX2P6WJ6BK4ODCYBRG6HISIFDBCSJL6A, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.242526026 (17m28.023459402s) [noise=true , address=45.91.116.4:4939 , piece_id=CHGWUUTASHFZYXQOXXWSEXRKNI7FVTRALPKD4TVZM5CGBRWNOWHA, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.244975691 (17m28.025909061s) [noise=true , address=45.91.116.5:4297 , piece_id=XCNIJJWIT3OMAK2GQJB7KBGAJHPHTL7GVWCCLCKDH2MK4DXWT6DQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.245067069 (17m28.026000444s) [noise=true , address=45.91.116.7:4011 , piece_id=2Q3VP4NNXV7FSIB3KTDKEI5VEA6FMJM3DV7WVXARUAAXIMVYPP5Q, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.245749945 (17m28.026683314s) [noise=true , address=45.91.116.8:4047 , piece_id=M6TIGG6K5IY5YLCWIIOAYM4OADHYT654KAHFZLAVHYRZFQXOMNSA, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.246361942 (17m28.02729532s) [noise=true , address=45.91.116.5:4299 , piece_id=MA22QRMEKYQGC4JHC73VHRZYUON6X5GSYCKPALUILHOTVE2JALZQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247224019 (17m28.028157391s) [noise=true , address=77.90.31.14:4663 , piece_id=4RJ57NHGYGNXNZ6WKJFSYTXABP5TCUW6SJ7N7BK6JYUI3CVXE6HQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247647098 (17m28.028580476s) [noise=true , address=45.91.116.10:4083 , piece_id=ULUVLWJPQ4GZASQJEH656LNH3LTDG437GOWBW2GCCVYALVSCLP2Q, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247771695 (17m28.028705064s) [noise=true , address=45.91.116.14:4190 , piece_id=YEFEQKDNF3RFW4YEUJG4CDDX2T4W5PYG2EBXZ6HOJTTGVD2L6LLQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247883972 (17m28.028817342s) [noise=true , address=45.91.116.9:4262 , piece_id=7ROQYHMFIZSJE3E5IBV2YYRFBOKXZFPP4APP64TLO7EO2XTQFN3A, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.248440176 (17m28.029373547s) [noise=true , address=45.91.116.7:4010 , piece_id=4TGUR3523QK6DZ7ZNPLQIGMN5L6TM5XXQFDTIPUBVUSNMUALMTXQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.291635996 (17m28.07256936s) [noise=true , address=77.90.31.16:4734 , piece_id=KBQPSDTMMERJGT3L4L7WH536IO7LESN3RBNGGHVGG7UM5ABQKBQQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:42.031234530 (17m57.812167916s) [noise=true , address=104.36.82.8:4970 , piece_id=I6WEZKRVPH3HR4HHMETUYYJSMHWRWXN4MU5YAZHTY5KPGIFY3GKA, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031315006 (17m57.812248388s) [noise=true , address=104.36.82.15:4258 , piece_id=GWV7CGQUBC6KBHB5CPSUAKGEXZIRV554OMJLZCMV2FNCYDRPIHIQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:41696->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031326322 (17m57.812259705s) [noise=true , address=104.36.82.7:4911 , piece_id=GFYIHUPEHJCF75CAFM4CZ7KUK4XD445KOF7UBVDM4HXMHRPWZYZQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:42482->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031335779 (17m57.812269142s) [noise=true , address=104.36.82.6:4873 , piece_id=PC4W673ITXSO7A6OR5XR6J6K562JMQHJUNEDLWPWH3JJOLDDXALQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031372559 (17m57.812305942s) [seg_pos={0 0}, upload=276]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:09:42.054859127 (17m57.835792503s) [upload=276]: Done uploading segments... Jul 26 22:09:42.054915101 (17m57.835848484s) [upload=276]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 802.057503ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 488.293404ms FAIL (2025-07-26 22:11:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 488.31313ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [...................................X..X........XXX.XXX.X.XX.] duration: 15.799016775s FAIL (2025-07-26 22:11:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 15.799036277s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [....................................X..X........XXX.XXX.X.XX] duration: 30.035424113s FAIL (2025-07-26 22:10:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.035463629s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 1m0.041053919s FAIL (2025-07-26 22:10:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== RLX7U7X2KNFJMQ3L72UUJMKHABDPLHBSU2GIJPH2MXDFAYYK6KOA, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.509371790 (18m4.29030519s) [noise=true , address=104.36.82.55:4206 , piece_id=KO5EEXOHCI6MX3SV63SK6PKMKZZZQ24OCD6M7KA3SADNWKAKTYBQ, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.509796952 (18m4.290730339s) [noise=true , address=104.36.82.54:4192 , piece_id=IIZ3D7YLCNOSGL36V3K22J6NULW3NWX7ZGAMJXHEG32CJXS7PJTA, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.510263762 (18m4.291197147s) [noise=true , address=104.36.82.53:4136 , piece_id=PNDHNF7GBQDVP5U7WAICEH3A2C3T7NNNRLBX7LJ725Q6EWX3YSAQ, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.510579816 (18m4.291513195s) [noise=true , address=79.127.219.51:4210 , piece_id=2SD4IHKDN4WYZSQXFE7ARYQT6FHIASQ7SHOKLP5LJUEH5SJ2LK6Q, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.510904245 (18m4.291837636s) [noise=true , address=104.36.82.52:4095 , piece_id=E5RQOLU23PV6XYT74D36J674TAPUCP6IGZJ2WJOJPFFZPZTAF7VA, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.511547775 (18m4.292481163s) [noise=true , address=104.36.82.46:4492 , piece_id=VBDK7XBDFGBFLLHAJ7UFHV5OHW3QDWQQIGTZB7Y6JJKAGDUUVU7A, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.522111824 (18m4.30304521s) [noise=true , address=79.127.219.38:4006 , piece_id=VWBPLOVGGFYWV3JZTT3PF4GANDPABKTUX3AF2MEUDE7LNVGIGVFA, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.522662154 (18m4.303595523s) [noise=true , address=79.127.205.234:4620 , piece_id=IOEQIBMU43W5XBOUY64KUAIWMRAURVWK6G5FUISHPYLATZGMXWSA, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.523108540 (18m4.304041923s) [noise=true , address=104.36.82.11:4124 , piece_id=43EAR24DLEWCH6TTLTOJOMTLIY77XL3772SE2JSJX54D5LB7JENQ, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.523507448 (18m4.304440837s) [noise=true , address=77.90.31.21:4887 , piece_id=LTR4YR37BTOEYNDN3RETTMGZVUEQXKNSH22EV7J5BXOYVKY7U5TQ, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.523905622 (18m4.30483901s) [noise=true , address=79.127.205.245:4244 , piece_id=GS7OAXCWFCUFHGU27BMY2AHJB7PI6PNJFPDNSVHGYK643AHGFLAA, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.524395675 (18m4.305329062s) [noise=true , address=104.36.82.47:4524 , piece_id=MS4XQQAGFJERE3RRBTO5OVRYU5N34PN35U7ZPXE23IXMOO7BJT7Q, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.524818300 (18m4.305751687s) [noise=true , address=104.36.82.4:4805 , piece_id=KMNOTZCYGEJUVDDCHNYGRHUUMPJP7MVN7KAKHXBCHKKYUI4WHO4Q, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.525230385 (18m4.306163772s) [noise=true , address=79.127.205.246:4279 , piece_id=B6M45DIWA475YFZPJVYO5WQKHTHSLLGC2E3NTPQZ6M26CBY6BQQA, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.525637032 (18m4.306570421s) [noise=true , address=104.36.82.49:4986 , piece_id=YIIKEX4YF4DC4NZ72VVT3NOGZP7TD6N4EDBHMXFUS4DQHQJFAKBQ, seg_pos={0 0}, upload=286]: Uploading piece... Jul 26 22:09:48.607432762 (18m4.388366145s) [upload=286]: Next returned nil segment Jul 26 22:09:48.607453899 (18m4.388387288s) [upload=286]: Waiting for error group managing segments... Jul 26 22:09:48.660583733 (18m4.441517112s) [noise=true , address=77.90.31.21:4887 , piece_id=LTR4YR37BTOEYNDN3RETTMGZVUEQXKNSH22EV7J5BXOYVKY7U5TQ, seg_pos={0 0}, upload=286]: Done uploading piece. err: <nil> Jul 26 22:09:48.861047113 (18m4.641980486s) [noise=true , address=79.127.219.38:4006 , piece_id=VWBPLOVGGFYWV3JZTT3PF4GANDPABKTUX3AF2MEUDE7LNVGIGVFA, seg_pos={0 0}, upload=286]: Done uploading piece. err: <nil> Jul 26 22:09:49.097533420 (18m4.878466791s) [noise=true , address=79.127.205.234:4620 , piece_id=IOEQIBMU43W5XBOUY64KUAIWMRAURVWK6G5FUISHPYLATZGMXWSA, seg_pos={0 0}, upload=286]: Done uploading piece. err: <nil> Jul 26 22:09:49.125802769 (18m4.906736132s) [noise=true , address=79.127.205.245:4244 , piece_id=GS7OAXCWFCUFHGU27BMY2AHJB7PI6PNJFPDNSVHGYK643AHGFLAA, seg_pos={0 0}, upload=286]: Done uploading piece. err: <nil> Jul 26 22:09:49.161926013 (18m4.942859375s) [noise=true , address=79.127.205.246:4279 , piece_id=B6M45DIWA475YFZPJVYO5WQKHTHSLLGC2E3NTPQZ6M26CBY6BQQA, seg_pos={0 0}, upload=286]: Done uploading piece. err: <nil> Jul 26 22:09:50.019160528 (18m5.800093892s) [noise=true , address=79.127.219.51:4210 , piece_id=2SD4IHKDN4WYZSQXFE7ARYQT6FHIASQ7SHOKLP5LJUEH5SJ2LK6Q, seg_pos={0 0}, upload=286]: Done uploading piece. err: <nil> Jul 26 22:09:50.436172344 (18m6.21710571s) [noise=true , address=64.49.13.170:4721 , piece_id=QEE3BVMD3BUME7JGV6SSVSSJRFR725L3G66KXJW2LM3HUR7ZX4OA, seg_pos={0 0}, upload=286]: Done uploading piece. err: <nil> Jul 26 22:10:48.426028794 (19m4.206962179s) [noise=true , address=104.36.82.55:4206 , piece_id=KO5EEXOHCI6MX3SV63SK6PKMKZZZQ24OCD6M7KA3SADNWKAKTYBQ, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:104.36.82.55:4206): protocol: rpc: dial tcp 104.36.82.55:4206: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426120841 (19m4.207054221s) [noise=true , address=104.36.82.54:4192 , piece_id=IIZ3D7YLCNOSGL36V3K22J6NULW3NWX7ZGAMJXHEG32CJXS7PJTA, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: rpc: dial tcp 104.36.82.54:4192: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426215409 (19m4.207148779s) [noise=true , address=104.36.82.46:4492 , piece_id=VBDK7XBDFGBFLLHAJ7UFHV5OHW3QDWQQIGTZB7Y6JJKAGDUUVU7A, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip, address:104.36.82.46:4492): protocol: rpc: dial tcp 104.36.82.46:4492: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426278846 (19m4.207212213s) [noise=true , address=104.36.82.52:4095 , piece_id=E5RQOLU23PV6XYT74D36J674TAPUCP6IGZJ2WJOJPFFZPZTAF7VA, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:1idvNc8SGRP7tkAckr2TLv7LnvftiRUTGcwy5GCrH5rVyDEK5N, address:104.36.82.52:4095): protocol: write tcp 159.223.143.39:46966->104.36.82.52:4095: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426356178 (19m4.207289545s) [noise=true , address=104.36.82.47:4524 , piece_id=MS4XQQAGFJERE3RRBTO5OVRYU5N34PN35U7ZPXE23IXMOO7BJT7Q, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: rpc: dial tcp 104.36.82.47:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426384348 (19m4.207317729s) [noise=true , address=104.36.82.40:4953 , piece_id=RLX7U7X2KNFJMQ3L72UUJMKHABDPLHBSU2GIJPH2MXDFAYYK6KOA, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:12dhNW98YkWKWzXBnwmTCvzaw9CtvxRW8ELJnAjHnATFhpM1oKn, address:104.36.82.40:4953): protocol: rpc: dial tcp 104.36.82.40:4953: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426412098 (19m4.207345468s) [noise=true , address=104.36.82.43:4863 , piece_id=5CU2NQSXCHI6UQI2L72NY7REKJZAHPDIJUOP3XMGBGTF466WISNQ, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:1hoK1e9x7ms16dWqXpNpfHA97U74iX9DkT7n73YUTcePgbMatZ, address:104.36.82.43:4863): protocol: rpc: dial tcp 104.36.82.43:4863: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426440839 (19m4.207374209s) [noise=true , address=104.36.82.11:4124 , piece_id=43EAR24DLEWCH6TTLTOJOMTLIY77XL3772SE2JSJX54D5LB7JENQ, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:1QAgfinEkxYwjzvtr53FRritkoDNbfYAw21qcJuj3uCY4JT4cZ, address:104.36.82.11:4124): protocol: rpc: dial tcp 104.36.82.11:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426468345 (19m4.207401729s) [noise=true , address=104.36.82.53:4136 , piece_id=PNDHNF7GBQDVP5U7WAICEH3A2C3T7NNNRLBX7LJ725Q6EWX3YSAQ, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:104.36.82.53:4136): protocol: rpc: dial tcp 104.36.82.53:4136: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426476674 (19m4.20741005s) [noise=true , address=104.36.82.4:4805 , piece_id=KMNOTZCYGEJUVDDCHNYGRHUUMPJP7MVN7KAKHXBCHKKYUI4WHO4Q, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:104.36.82.4:4805): protocol: rpc: dial tcp 104.36.82.4:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426535131 (19m4.207468507s) [noise=true , address=104.36.82.49:4986 , piece_id=YIIKEX4YF4DC4NZ72VVT3NOGZP7TD6N4EDBHMXFUS4DQHQJFAKBQ, seg_pos={0 0}, upload=286]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 159.223.143.39:55508->104.36.82.49:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:10:48.426581525 (19m4.207514922s) [seg_pos={0 0}, upload=286]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:10:48.464956953 (19m4.245890344s) [upload=286]: Done uploading segments... Jul 26 22:10:48.464988777 (19m4.245922164s) [upload=286]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.041072378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 507.370138ms FAIL (2025-07-26 22:10:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 507.443577ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...................................XX..XXX..X..XX..X..XX.X..] duration: 389.510104ms FAIL (2025-07-26 22:10:38 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:22.534459505 (16m38.315392869s) [noise=true , address=104.36.82.3:4765 , piece_id=QY5IWCE4YCDVGCA54ZNJR5CNH5POOREICCZCM6SNGI3I3QPZPTWQ, seg_pos={0 0}, upload=263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: write tcp 159.223.143.39:34250->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:22.534491171 (16m38.315424558s) [seg_pos={0 0}, upload=263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:08:22.534577358 (16m38.315510743s) [upload=263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:08:22.534582422 (16m38.315515801s) [upload=263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:08:22.575438889 (16m38.356372275s) [upload=263]: Done uploading segments... Jul 26 22:08:22.575462290 (16m38.356395666s) [upload=263]: Upload finished. err: <nil> Jul 26 22:09:12.073816930 (17m27.854750309s) [upload=276]: Starting upload Jul 26 22:09:12.073908198 (17m27.854841586s) [upload=276]: Done starting upload Jul 26 22:09:12.074014224 (17m27.854947604s) [upload=276]: Uploading segments... Jul 26 22:09:12.074306818 (17m27.8552402s) [upload=276]: Got next segment. Inline: false Jul 26 22:09:12.074373049 (17m27.855306427s) [upload=276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:09:12.074386765 (17m27.855320186s) [upload=276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:09:12.102208192 (17m27.883141579s) [seg_pos={0 0}, upload=276]: Begin upload segment... Jul 26 22:09:12.102281970 (17m27.883215358s) [seg_pos={0 0}, upload=276]: Done begin upload segment. Jul 26 22:09:12.102316815 (17m27.883250203s) [noise=true , address=104.36.82.6:4873 , piece_id=PC4W673ITXSO7A6OR5XR6J6K562JMQHJUNEDLWPWH3JJOLDDXALQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.103398160 (17m27.884331551s) [noise=true , address=45.91.116.5:4297 , piece_id=XCNIJJWIT3OMAK2GQJB7KBGAJHPHTL7GVWCCLCKDH2MK4DXWT6DQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.103834590 (17m27.884767971s) [noise=true , address=104.36.82.8:4970 , piece_id=I6WEZKRVPH3HR4HHMETUYYJSMHWRWXN4MU5YAZHTY5KPGIFY3GKA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.104321474 (17m27.885254862s) [noise=true , address=77.90.31.18:4807 , piece_id=KC253KPUC73VN7SPMTJZUX2P6WJ6BK4ODCYBRG6HISIFDBCSJL6A, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.105886362 (17m27.886819751s) [noise=true , address=45.91.116.4:4939 , piece_id=CHGWUUTASHFZYXQOXXWSEXRKNI7FVTRALPKD4TVZM5CGBRWNOWHA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.106338669 (17m27.887272047s) [noise=true , address=77.90.31.14:4663 , piece_id=4RJ57NHGYGNXNZ6WKJFSYTXABP5TCUW6SJ7N7BK6JYUI3CVXE6HQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.106682458 (17m27.887615847s) [noise=true , address=104.36.82.15:4258 , piece_id=GWV7CGQUBC6KBHB5CPSUAKGEXZIRV554OMJLZCMV2FNCYDRPIHIQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.107171678 (17m27.888105069s) [noise=true , address=77.90.31.16:4734 , piece_id=KBQPSDTMMERJGT3L4L7WH536IO7LESN3RBNGGHVGG7UM5ABQKBQQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.107598895 (17m27.888532281s) [noise=true , address=45.91.116.5:4299 , piece_id=MA22QRMEKYQGC4JHC73VHRZYUON6X5GSYCKPALUILHOTVE2JALZQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.110879103 (17m27.891812491s) [noise=true , address=45.91.116.10:4083 , piece_id=ULUVLWJPQ4GZASQJEH656LNH3LTDG437GOWBW2GCCVYALVSCLP2Q, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111179084 (17m27.892112471s) [noise=true , address=45.91.116.14:4190 , piece_id=YEFEQKDNF3RFW4YEUJG4CDDX2T4W5PYG2EBXZ6HOJTTGVD2L6LLQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111561235 (17m27.892494622s) [noise=true , address=45.91.116.7:4011 , piece_id=2Q3VP4NNXV7FSIB3KTDKEI5VEA6FMJM3DV7WVXARUAAXIMVYPP5Q, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111944292 (17m27.892877681s) [noise=true , address=104.36.82.7:4911 , piece_id=GFYIHUPEHJCF75CAFM4CZ7KUK4XD445KOF7UBVDM4HXMHRPWZYZQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.112357623 (17m27.893291011s) [noise=true , address=45.91.116.8:4047 , piece_id=M6TIGG6K5IY5YLCWIIOAYM4OADHYT654KAHFZLAVHYRZFQXOMNSA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.112763233 (17m27.893696631s) [noise=true , address=45.91.116.9:4262 , piece_id=7ROQYHMFIZSJE3E5IBV2YYRFBOKXZFPP4APP64TLO7EO2XTQFN3A, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.113189270 (17m27.894122659s) [noise=true , address=45.91.116.7:4010 , piece_id=4TGUR3523QK6DZ7ZNPLQIGMN5L6TM5XXQFDTIPUBVUSNMUALMTXQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.237315416 (17m28.018248798s) [upload=276]: Next returned nil segment Jul 26 22:09:12.237338503 (17m28.018271879s) [upload=276]: Waiting for error group managing segments... Jul 26 22:09:12.242491332 (17m28.023424712s) [noise=true , address=77.90.31.18:4807 , piece_id=KC253KPUC73VN7SPMTJZUX2P6WJ6BK4ODCYBRG6HISIFDBCSJL6A, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.242526026 (17m28.023459402s) [noise=true , address=45.91.116.4:4939 , piece_id=CHGWUUTASHFZYXQOXXWSEXRKNI7FVTRALPKD4TVZM5CGBRWNOWHA, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.244975691 (17m28.025909061s) [noise=true , address=45.91.116.5:4297 , piece_id=XCNIJJWIT3OMAK2GQJB7KBGAJHPHTL7GVWCCLCKDH2MK4DXWT6DQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.245067069 (17m28.026000444s) [noise=true , address=45.91.116.7:4011 , piece_id=2Q3VP4NNXV7FSIB3KTDKEI5VEA6FMJM3DV7WVXARUAAXIMVYPP5Q, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.245749945 (17m28.026683314s) [noise=true , address=45.91.116.8:4047 , piece_id=M6TIGG6K5IY5YLCWIIOAYM4OADHYT654KAHFZLAVHYRZFQXOMNSA, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.246361942 (17m28.02729532s) [noise=true , address=45.91.116.5:4299 , piece_id=MA22QRMEKYQGC4JHC73VHRZYUON6X5GSYCKPALUILHOTVE2JALZQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247224019 (17m28.028157391s) [noise=true , address=77.90.31.14:4663 , piece_id=4RJ57NHGYGNXNZ6WKJFSYTXABP5TCUW6SJ7N7BK6JYUI3CVXE6HQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247647098 (17m28.028580476s) [noise=true , address=45.91.116.10:4083 , piece_id=ULUVLWJPQ4GZASQJEH656LNH3LTDG437GOWBW2GCCVYALVSCLP2Q, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247771695 (17m28.028705064s) [noise=true , address=45.91.116.14:4190 , piece_id=YEFEQKDNF3RFW4YEUJG4CDDX2T4W5PYG2EBXZ6HOJTTGVD2L6LLQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247883972 (17m28.028817342s) [noise=true , address=45.91.116.9:4262 , piece_id=7ROQYHMFIZSJE3E5IBV2YYRFBOKXZFPP4APP64TLO7EO2XTQFN3A, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.248440176 (17m28.029373547s) [noise=true , address=45.91.116.7:4010 , piece_id=4TGUR3523QK6DZ7ZNPLQIGMN5L6TM5XXQFDTIPUBVUSNMUALMTXQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.291635996 (17m28.07256936s) [noise=true , address=77.90.31.16:4734 , piece_id=KBQPSDTMMERJGT3L4L7WH536IO7LESN3RBNGGHVGG7UM5ABQKBQQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:42.031234530 (17m57.812167916s) [noise=true , address=104.36.82.8:4970 , piece_id=I6WEZKRVPH3HR4HHMETUYYJSMHWRWXN4MU5YAZHTY5KPGIFY3GKA, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031315006 (17m57.812248388s) [noise=true , address=104.36.82.15:4258 , piece_id=GWV7CGQUBC6KBHB5CPSUAKGEXZIRV554OMJLZCMV2FNCYDRPIHIQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:41696->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031326322 (17m57.812259705s) [noise=true , address=104.36.82.7:4911 , piece_id=GFYIHUPEHJCF75CAFM4CZ7KUK4XD445KOF7UBVDM4HXMHRPWZYZQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:42482->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031335779 (17m57.812269142s) [noise=true , address=104.36.82.6:4873 , piece_id=PC4W673ITXSO7A6OR5XR6J6K562JMQHJUNEDLWPWH3JJOLDDXALQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031372559 (17m57.812305942s) [seg_pos={0 0}, upload=276]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:09:42.054859127 (17m57.835792503s) [upload=276]: Done uploading segments... Jul 26 22:09:42.054915101 (17m57.835848484s) [upload=276]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 389.5444ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 19.424722685s FAIL (2025-07-26 22:10:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.424751519s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....................................X..X........XXX.XXX.X.X] duration: 30.03749225s FAIL (2025-07-26 22:10:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.037525785s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....................................XX..XXX..X..XX..X..XX.X.] duration: 486.91647ms FAIL (2025-07-26 22:09:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:22.534459505 (16m38.315392869s) [noise=true , address=104.36.82.3:4765 , piece_id=QY5IWCE4YCDVGCA54ZNJR5CNH5POOREICCZCM6SNGI3I3QPZPTWQ, seg_pos={0 0}, upload=263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: write tcp 159.223.143.39:34250->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:22.534491171 (16m38.315424558s) [seg_pos={0 0}, upload=263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:08:22.534577358 (16m38.315510743s) [upload=263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:08:22.534582422 (16m38.315515801s) [upload=263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:08:22.575438889 (16m38.356372275s) [upload=263]: Done uploading segments... Jul 26 22:08:22.575462290 (16m38.356395666s) [upload=263]: Upload finished. err: <nil> Jul 26 22:09:12.073816930 (17m27.854750309s) [upload=276]: Starting upload Jul 26 22:09:12.073908198 (17m27.854841586s) [upload=276]: Done starting upload Jul 26 22:09:12.074014224 (17m27.854947604s) [upload=276]: Uploading segments... Jul 26 22:09:12.074306818 (17m27.8552402s) [upload=276]: Got next segment. Inline: false Jul 26 22:09:12.074373049 (17m27.855306427s) [upload=276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:09:12.074386765 (17m27.855320186s) [upload=276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:09:12.102208192 (17m27.883141579s) [seg_pos={0 0}, upload=276]: Begin upload segment... Jul 26 22:09:12.102281970 (17m27.883215358s) [seg_pos={0 0}, upload=276]: Done begin upload segment. Jul 26 22:09:12.102316815 (17m27.883250203s) [noise=true , address=104.36.82.6:4873 , piece_id=PC4W673ITXSO7A6OR5XR6J6K562JMQHJUNEDLWPWH3JJOLDDXALQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.103398160 (17m27.884331551s) [noise=true , address=45.91.116.5:4297 , piece_id=XCNIJJWIT3OMAK2GQJB7KBGAJHPHTL7GVWCCLCKDH2MK4DXWT6DQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.103834590 (17m27.884767971s) [noise=true , address=104.36.82.8:4970 , piece_id=I6WEZKRVPH3HR4HHMETUYYJSMHWRWXN4MU5YAZHTY5KPGIFY3GKA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.104321474 (17m27.885254862s) [noise=true , address=77.90.31.18:4807 , piece_id=KC253KPUC73VN7SPMTJZUX2P6WJ6BK4ODCYBRG6HISIFDBCSJL6A, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.105886362 (17m27.886819751s) [noise=true , address=45.91.116.4:4939 , piece_id=CHGWUUTASHFZYXQOXXWSEXRKNI7FVTRALPKD4TVZM5CGBRWNOWHA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.106338669 (17m27.887272047s) [noise=true , address=77.90.31.14:4663 , piece_id=4RJ57NHGYGNXNZ6WKJFSYTXABP5TCUW6SJ7N7BK6JYUI3CVXE6HQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.106682458 (17m27.887615847s) [noise=true , address=104.36.82.15:4258 , piece_id=GWV7CGQUBC6KBHB5CPSUAKGEXZIRV554OMJLZCMV2FNCYDRPIHIQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.107171678 (17m27.888105069s) [noise=true , address=77.90.31.16:4734 , piece_id=KBQPSDTMMERJGT3L4L7WH536IO7LESN3RBNGGHVGG7UM5ABQKBQQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.107598895 (17m27.888532281s) [noise=true , address=45.91.116.5:4299 , piece_id=MA22QRMEKYQGC4JHC73VHRZYUON6X5GSYCKPALUILHOTVE2JALZQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.110879103 (17m27.891812491s) [noise=true , address=45.91.116.10:4083 , piece_id=ULUVLWJPQ4GZASQJEH656LNH3LTDG437GOWBW2GCCVYALVSCLP2Q, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111179084 (17m27.892112471s) [noise=true , address=45.91.116.14:4190 , piece_id=YEFEQKDNF3RFW4YEUJG4CDDX2T4W5PYG2EBXZ6HOJTTGVD2L6LLQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111561235 (17m27.892494622s) [noise=true , address=45.91.116.7:4011 , piece_id=2Q3VP4NNXV7FSIB3KTDKEI5VEA6FMJM3DV7WVXARUAAXIMVYPP5Q, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111944292 (17m27.892877681s) [noise=true , address=104.36.82.7:4911 , piece_id=GFYIHUPEHJCF75CAFM4CZ7KUK4XD445KOF7UBVDM4HXMHRPWZYZQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.112357623 (17m27.893291011s) [noise=true , address=45.91.116.8:4047 , piece_id=M6TIGG6K5IY5YLCWIIOAYM4OADHYT654KAHFZLAVHYRZFQXOMNSA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.112763233 (17m27.893696631s) [noise=true , address=45.91.116.9:4262 , piece_id=7ROQYHMFIZSJE3E5IBV2YYRFBOKXZFPP4APP64TLO7EO2XTQFN3A, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.113189270 (17m27.894122659s) [noise=true , address=45.91.116.7:4010 , piece_id=4TGUR3523QK6DZ7ZNPLQIGMN5L6TM5XXQFDTIPUBVUSNMUALMTXQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.237315416 (17m28.018248798s) [upload=276]: Next returned nil segment Jul 26 22:09:12.237338503 (17m28.018271879s) [upload=276]: Waiting for error group managing segments... Jul 26 22:09:12.242491332 (17m28.023424712s) [noise=true , address=77.90.31.18:4807 , piece_id=KC253KPUC73VN7SPMTJZUX2P6WJ6BK4ODCYBRG6HISIFDBCSJL6A, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.242526026 (17m28.023459402s) [noise=true , address=45.91.116.4:4939 , piece_id=CHGWUUTASHFZYXQOXXWSEXRKNI7FVTRALPKD4TVZM5CGBRWNOWHA, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.244975691 (17m28.025909061s) [noise=true , address=45.91.116.5:4297 , piece_id=XCNIJJWIT3OMAK2GQJB7KBGAJHPHTL7GVWCCLCKDH2MK4DXWT6DQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.245067069 (17m28.026000444s) [noise=true , address=45.91.116.7:4011 , piece_id=2Q3VP4NNXV7FSIB3KTDKEI5VEA6FMJM3DV7WVXARUAAXIMVYPP5Q, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.245749945 (17m28.026683314s) [noise=true , address=45.91.116.8:4047 , piece_id=M6TIGG6K5IY5YLCWIIOAYM4OADHYT654KAHFZLAVHYRZFQXOMNSA, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.246361942 (17m28.02729532s) [noise=true , address=45.91.116.5:4299 , piece_id=MA22QRMEKYQGC4JHC73VHRZYUON6X5GSYCKPALUILHOTVE2JALZQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247224019 (17m28.028157391s) [noise=true , address=77.90.31.14:4663 , piece_id=4RJ57NHGYGNXNZ6WKJFSYTXABP5TCUW6SJ7N7BK6JYUI3CVXE6HQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247647098 (17m28.028580476s) [noise=true , address=45.91.116.10:4083 , piece_id=ULUVLWJPQ4GZASQJEH656LNH3LTDG437GOWBW2GCCVYALVSCLP2Q, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247771695 (17m28.028705064s) [noise=true , address=45.91.116.14:4190 , piece_id=YEFEQKDNF3RFW4YEUJG4CDDX2T4W5PYG2EBXZ6HOJTTGVD2L6LLQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247883972 (17m28.028817342s) [noise=true , address=45.91.116.9:4262 , piece_id=7ROQYHMFIZSJE3E5IBV2YYRFBOKXZFPP4APP64TLO7EO2XTQFN3A, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.248440176 (17m28.029373547s) [noise=true , address=45.91.116.7:4010 , piece_id=4TGUR3523QK6DZ7ZNPLQIGMN5L6TM5XXQFDTIPUBVUSNMUALMTXQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.291635996 (17m28.07256936s) [noise=true , address=77.90.31.16:4734 , piece_id=KBQPSDTMMERJGT3L4L7WH536IO7LESN3RBNGGHVGG7UM5ABQKBQQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:42.031234530 (17m57.812167916s) [noise=true , address=104.36.82.8:4970 , piece_id=I6WEZKRVPH3HR4HHMETUYYJSMHWRWXN4MU5YAZHTY5KPGIFY3GKA, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031315006 (17m57.812248388s) [noise=true , address=104.36.82.15:4258 , piece_id=GWV7CGQUBC6KBHB5CPSUAKGEXZIRV554OMJLZCMV2FNCYDRPIHIQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:41696->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031326322 (17m57.812259705s) [noise=true , address=104.36.82.7:4911 , piece_id=GFYIHUPEHJCF75CAFM4CZ7KUK4XD445KOF7UBVDM4HXMHRPWZYZQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:42482->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031335779 (17m57.812269142s) [noise=true , address=104.36.82.6:4873 , piece_id=PC4W673ITXSO7A6OR5XR6J6K562JMQHJUNEDLWPWH3JJOLDDXALQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031372559 (17m57.812305942s) [seg_pos={0 0}, upload=276]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:09:42.054859127 (17m57.835792503s) [upload=276]: Done uploading segments... Jul 26 22:09:42.054915101 (17m57.835848484s) [upload=276]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 486.956049ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 1m0.043322661s FAIL (2025-07-26 22:09:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 08:48.400704778 (17m4.181638164s) [noise=true , address=79.127.201.215:4358 , piece_id=PVQ4HVTXHAMFOII7QNQKFTJN5IOVTVE5ATU4XT4ON7Z53VQXU5XA, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.401251811 (17m4.182185199s) [noise=true , address=104.36.82.15:4264 , piece_id=TCGRSKJU43FWEJ72HEWM4ZTBEVOL522OYSK54YZVZZEDTVVFCKBA, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.402086322 (17m4.183019707s) [seg_pos={0 0}, upload=269]: Done begin upload segment. Jul 26 22:08:48.402144413 (17m4.1830778s) [noise=true , address=45.91.116.13:4187 , piece_id=CAGAOV7V7C65VPT6WRY7YVNWR7WVEYMRMNHU2NI2B66CG3PNS3VA, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.403414141 (17m4.184347528s) [noise=true , address=79.127.219.47:4061 , piece_id=NNOGEFSBR7FR6GT6LYW3PLGA3O4JJMUV5CSXENNAXBQRERUC2SAA, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.403932677 (17m4.184866064s) [noise=true , address=79.127.205.245:4223 , piece_id=GJ4KDQSZ5PV56T77NUA6EQ26GGMYLZLPSRNZSHVNDTRZ4LB22DEA, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.404673607 (17m4.185606997s) [noise=true , address=104.36.82.19:4423 , piece_id=3HJUTQLWJYQJEODGYQN3EE3TCEIJUISCMYAAXZ42LCAY6N5PT4EQ, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.405101497 (17m4.186034886s) [noise=true , address=79.127.219.51:4210 , piece_id=6WHNWTAMU2HETLFQG6BZWKCA74Y5XGI4DOPA3YK2WQEJ54FLSLRQ, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.405660116 (17m4.186593507s) [noise=true , address=104.36.82.9:4015 , piece_id=QTCRZXFAL7QI3OVIDNGVHIORARTY3KXRYFLKAWSU6E774HVBK52Q, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.406663555 (17m4.18759694s) [noise=true , address=79.127.219.38:4004 , piece_id=LBDKQYAWJTL5RZV7M7AWDYC6SMM7V7S25RRYMXAP5RVY7F2XFJQA, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.407956920 (17m4.188890309s) [noise=true , address=104.36.82.42:4824 , piece_id=EKW3KQRCKV5XYXBZWWNZ6HDL4B3PDRP73TQ6D54ZF4Q6EFN464KQ, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.409089078 (17m4.190022466s) [noise=true , address=104.36.82.47:4534 , piece_id=ML4DM3CV5UIWIJKIQJIQ2GV5U3IHOW7JNWVUDR5PDJNHMAFRTQWA, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.409711922 (17m4.190645301s) [noise=true , address=64.49.13.138:4483 , piece_id=FHA5TNIDN5I2IT7BNJURK3W4RZUSGQUIELEW5OMGFDCXVBBKLPMQ, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.410687163 (17m4.191620546s) [noise=true , address=104.36.82.46:4507 , piece_id=LAEJTLITKEQQ7OB64YUKINFUPSCABGO7GBZQ424644ET72HA764Q, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.411272997 (17m4.192206382s) [noise=true , address=77.90.31.10:4528 , piece_id=52FQJ7NZE6FH5G32UQTPW2HLWADFQLHBAUBB2IIT32FTB23RDGLA, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.411783487 (17m4.192716874s) [noise=true , address=104.36.82.56:4262 , piece_id=LXJK4IFOR4JR7WSVUM72ZF672Z2KCMVTYUNORAIEN3JSB3YUDXUQ, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.413031789 (17m4.193965184s) [noise=true , address=104.36.82.4:4836 , piece_id=M5MPJYV5ZGWG5U6OMFWY6EUGXT3G45KQ33XLFXY5SJNEY2EZ3LLQ, seg_pos={0 0}, upload=269]: Uploading piece... Jul 26 22:08:48.542269526 (17m4.323202904s) [upload=269]: Next returned nil segment Jul 26 22:08:48.542290707 (17m4.323224096s) [upload=269]: Waiting for error group managing segments... Jul 26 22:08:49.072013021 (17m4.852946385s) [noise=true , address=79.127.205.245:4223 , piece_id=GJ4KDQSZ5PV56T77NUA6EQ26GGMYLZLPSRNZSHVNDTRZ4LB22DEA, seg_pos={0 0}, upload=269]: Done uploading piece. err: <nil> Jul 26 22:08:49.186052455 (17m4.966985831s) [noise=true , address=45.91.116.13:4187 , piece_id=CAGAOV7V7C65VPT6WRY7YVNWR7WVEYMRMNHU2NI2B66CG3PNS3VA, seg_pos={0 0}, upload=269]: Done uploading piece. err: <nil> Jul 26 22:08:49.222982791 (17m5.003916159s) [noise=true , address=64.49.13.138:4483 , piece_id=FHA5TNIDN5I2IT7BNJURK3W4RZUSGQUIELEW5OMGFDCXVBBKLPMQ, seg_pos={0 0}, upload=269]: Done uploading piece. err: <nil> Jul 26 22:08:50.728354087 (17m6.509287463s) [noise=true , address=79.127.219.51:4210 , piece_id=6WHNWTAMU2HETLFQG6BZWKCA74Y5XGI4DOPA3YK2WQEJ54FLSLRQ, seg_pos={0 0}, upload=269]: Done uploading piece. err: <nil> Jul 26 22:08:53.108813334 (17m8.889746719s) [noise=true , address=79.127.219.47:4061 , piece_id=NNOGEFSBR7FR6GT6LYW3PLGA3O4JJMUV5CSXENNAXBQRERUC2SAA, seg_pos={0 0}, upload=269]: Done uploading piece. err: <nil> Jul 26 22:08:53.934351704 (17m9.715285068s) [noise=true , address=77.90.31.13:4659 , piece_id=JNDCWIRSZGLFKYZEDVWN3QD5Y6IVIBNL3P2VZRYQUPB4SH23KEHQ, seg_pos={0 0}, upload=269]: Done uploading piece. err: <nil> Jul 26 22:08:56.135060843 (17m11.915994206s) [noise=true , address=79.127.219.38:4004 , piece_id=LBDKQYAWJTL5RZV7M7AWDYC6SMM7V7S25RRYMXAP5RVY7F2XFJQA, seg_pos={0 0}, upload=269]: Done uploading piece. err: <nil> Jul 26 22:09:01.914719573 (17m17.695652948s) [noise=true , address=79.127.201.215:4358 , piece_id=PVQ4HVTXHAMFOII7QNQKFTJN5IOVTVE5ATU4XT4ON7Z53VQXU5XA, seg_pos={0 0}, upload=269]: Done uploading piece. err: <nil> Jul 26 22:09:48.317195324 (18m4.098128705s) [noise=true , address=104.36.82.19:4423 , piece_id=3HJUTQLWJYQJEODGYQN3EE3TCEIJUISCMYAAXZ42LCAY6N5PT4EQ, seg_pos={0 0}, upload=269]: Done uploading piece. err: ecclient: upload failed (node:12Rpox27kwUgBKkqhfexgNQPEomMivnFFy5bJAmNmbaFDhaaBSM, address:104.36.82.19:4423): protocol: write tcp 159.223.143.39:38224->104.36.82.19:4423: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:48.317289726 (18m4.098223111s) [noise=true , address=104.36.82.4:4836 , piece_id=M5MPJYV5ZGWG5U6OMFWY6EUGXT3G45KQ33XLFXY5SJNEY2EZ3LLQ, seg_pos={0 0}, upload=269]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:104.36.82.4:4836): protocol: write tcp 159.223.143.39:58206->104.36.82.4:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:48.317360436 (18m4.098293816s) [noise=true , address=104.36.82.42:4824 , piece_id=EKW3KQRCKV5XYXBZWWNZ6HDL4B3PDRP73TQ6D54ZF4Q6EFN464KQ, seg_pos={0 0}, upload=269]: Done uploading piece. err: ecclient: upload failed (node:12QTjnomQYFETHbaJBWxWEPo5yweArk68bKPeooZXpRZrpw1ACq, address:104.36.82.42:4824): protocol: rpc: dial tcp 104.36.82.42:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:48.317424850 (18m4.098358227s) [noise=true , address=104.36.82.47:4534 , piece_id=ML4DM3CV5UIWIJKIQJIQ2GV5U3IHOW7JNWVUDR5PDJNHMAFRTQWA, seg_pos={0 0}, upload=269]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: rpc: dial tcp 104.36.82.47:4534: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:48.317492277 (18m4.098425654s) [noise=true , address=104.36.82.46:4507 , piece_id=LAEJTLITKEQQ7OB64YUKINFUPSCABGO7GBZQ424644ET72HA764Q, seg_pos={0 0}, upload=269]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: rpc: dial tcp 104.36.82.46:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:48.317507975 (18m4.098441338s) [noise=true , address=104.36.82.15:4264 , piece_id=TCGRSKJU43FWEJ72HEWM4ZTBEVOL522OYSK54YZVZZEDTVVFCKBA, seg_pos={0 0}, upload=269]: Done uploading piece. err: ecclient: upload failed (node:1ovYsPwYdx7AYPboFx3Xdcg4VQrPGZj2XTMTqgox6zm81gbf8Z, address:104.36.82.15:4264): protocol: write tcp 159.223.143.39:55058->104.36.82.15:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:48.317540509 (18m4.098473888s) [noise=true , address=77.90.31.10:4528 , piece_id=52FQJ7NZE6FH5G32UQTPW2HLWADFQLHBAUBB2IIT32FTB23RDGLA, seg_pos={0 0}, upload=269]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: rpc: dial tcp 77.90.31.10:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:48.317606205 (18m4.098539583s) [noise=true , address=104.36.82.56:4262 , piece_id=LXJK4IFOR4JR7WSVUM72ZF672Z2KCMVTYUNORAIEN3JSB3YUDXUQ, seg_pos={0 0}, upload=269]: Done uploading piece. err: ecclient: upload failed (node:1DY2kpdCo8EDSs1LHL6qBFo6qjnsb7FL1QDekMV5qaWB5jxeMz, address:104.36.82.56:4262): protocol: rpc: dial tcp 104.36.82.56:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:48.317632066 (18m4.098565436s) [noise=true , address=104.36.82.9:4015 , piece_id=QTCRZXFAL7QI3OVIDNGVHIORARTY3KXRYFLKAWSU6E774HVBK52Q, seg_pos={0 0}, upload=269]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: rpc: dial tcp 104.36.82.9:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:48.317682739 (18m4.098616116s) [noise=true , address=104.36.82.55:4199 , piece_id=U2XCVUT67ZKCDUJLKIOVIYGDIWBAW3QNNANE7LUP6ISFWORLYPVQ, seg_pos={0 0}, upload=269]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: rpc: dial tcp 104.36.82.55:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:48.317744317 (18m4.098677704s) [seg_pos={0 0}, upload=269]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:09:48.357963129 (18m4.138896518s) [upload=269]: Done uploading segments... Jul 26 22:09:48.357999274 (18m4.138932657s) [upload=269]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.043358385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....................................XX..XXX..X..XX..X..XX.X] duration: 30.026493819s FAIL (2025-07-26 22:09:42 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:22.534459505 (16m38.315392869s) [noise=true , address=104.36.82.3:4765 , piece_id=QY5IWCE4YCDVGCA54ZNJR5CNH5POOREICCZCM6SNGI3I3QPZPTWQ, seg_pos={0 0}, upload=263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: write tcp 159.223.143.39:34250->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:22.534491171 (16m38.315424558s) [seg_pos={0 0}, upload=263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:08:22.534577358 (16m38.315510743s) [upload=263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:08:22.534582422 (16m38.315515801s) [upload=263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:08:22.575438889 (16m38.356372275s) [upload=263]: Done uploading segments... Jul 26 22:08:22.575462290 (16m38.356395666s) [upload=263]: Upload finished. err: <nil> Jul 26 22:09:12.073816930 (17m27.854750309s) [upload=276]: Starting upload Jul 26 22:09:12.073908198 (17m27.854841586s) [upload=276]: Done starting upload Jul 26 22:09:12.074014224 (17m27.854947604s) [upload=276]: Uploading segments... Jul 26 22:09:12.074306818 (17m27.8552402s) [upload=276]: Got next segment. Inline: false Jul 26 22:09:12.074373049 (17m27.855306427s) [upload=276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:09:12.074386765 (17m27.855320186s) [upload=276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:09:12.102208192 (17m27.883141579s) [seg_pos={0 0}, upload=276]: Begin upload segment... Jul 26 22:09:12.102281970 (17m27.883215358s) [seg_pos={0 0}, upload=276]: Done begin upload segment. Jul 26 22:09:12.102316815 (17m27.883250203s) [noise=true , address=104.36.82.6:4873 , piece_id=PC4W673ITXSO7A6OR5XR6J6K562JMQHJUNEDLWPWH3JJOLDDXALQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.103398160 (17m27.884331551s) [noise=true , address=45.91.116.5:4297 , piece_id=XCNIJJWIT3OMAK2GQJB7KBGAJHPHTL7GVWCCLCKDH2MK4DXWT6DQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.103834590 (17m27.884767971s) [noise=true , address=104.36.82.8:4970 , piece_id=I6WEZKRVPH3HR4HHMETUYYJSMHWRWXN4MU5YAZHTY5KPGIFY3GKA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.104321474 (17m27.885254862s) [noise=true , address=77.90.31.18:4807 , piece_id=KC253KPUC73VN7SPMTJZUX2P6WJ6BK4ODCYBRG6HISIFDBCSJL6A, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.105886362 (17m27.886819751s) [noise=true , address=45.91.116.4:4939 , piece_id=CHGWUUTASHFZYXQOXXWSEXRKNI7FVTRALPKD4TVZM5CGBRWNOWHA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.106338669 (17m27.887272047s) [noise=true , address=77.90.31.14:4663 , piece_id=4RJ57NHGYGNXNZ6WKJFSYTXABP5TCUW6SJ7N7BK6JYUI3CVXE6HQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.106682458 (17m27.887615847s) [noise=true , address=104.36.82.15:4258 , piece_id=GWV7CGQUBC6KBHB5CPSUAKGEXZIRV554OMJLZCMV2FNCYDRPIHIQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.107171678 (17m27.888105069s) [noise=true , address=77.90.31.16:4734 , piece_id=KBQPSDTMMERJGT3L4L7WH536IO7LESN3RBNGGHVGG7UM5ABQKBQQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.107598895 (17m27.888532281s) [noise=true , address=45.91.116.5:4299 , piece_id=MA22QRMEKYQGC4JHC73VHRZYUON6X5GSYCKPALUILHOTVE2JALZQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.110879103 (17m27.891812491s) [noise=true , address=45.91.116.10:4083 , piece_id=ULUVLWJPQ4GZASQJEH656LNH3LTDG437GOWBW2GCCVYALVSCLP2Q, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111179084 (17m27.892112471s) [noise=true , address=45.91.116.14:4190 , piece_id=YEFEQKDNF3RFW4YEUJG4CDDX2T4W5PYG2EBXZ6HOJTTGVD2L6LLQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111561235 (17m27.892494622s) [noise=true , address=45.91.116.7:4011 , piece_id=2Q3VP4NNXV7FSIB3KTDKEI5VEA6FMJM3DV7WVXARUAAXIMVYPP5Q, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.111944292 (17m27.892877681s) [noise=true , address=104.36.82.7:4911 , piece_id=GFYIHUPEHJCF75CAFM4CZ7KUK4XD445KOF7UBVDM4HXMHRPWZYZQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.112357623 (17m27.893291011s) [noise=true , address=45.91.116.8:4047 , piece_id=M6TIGG6K5IY5YLCWIIOAYM4OADHYT654KAHFZLAVHYRZFQXOMNSA, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.112763233 (17m27.893696631s) [noise=true , address=45.91.116.9:4262 , piece_id=7ROQYHMFIZSJE3E5IBV2YYRFBOKXZFPP4APP64TLO7EO2XTQFN3A, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.113189270 (17m27.894122659s) [noise=true , address=45.91.116.7:4010 , piece_id=4TGUR3523QK6DZ7ZNPLQIGMN5L6TM5XXQFDTIPUBVUSNMUALMTXQ, seg_pos={0 0}, upload=276]: Uploading piece... Jul 26 22:09:12.237315416 (17m28.018248798s) [upload=276]: Next returned nil segment Jul 26 22:09:12.237338503 (17m28.018271879s) [upload=276]: Waiting for error group managing segments... Jul 26 22:09:12.242491332 (17m28.023424712s) [noise=true , address=77.90.31.18:4807 , piece_id=KC253KPUC73VN7SPMTJZUX2P6WJ6BK4ODCYBRG6HISIFDBCSJL6A, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.242526026 (17m28.023459402s) [noise=true , address=45.91.116.4:4939 , piece_id=CHGWUUTASHFZYXQOXXWSEXRKNI7FVTRALPKD4TVZM5CGBRWNOWHA, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.244975691 (17m28.025909061s) [noise=true , address=45.91.116.5:4297 , piece_id=XCNIJJWIT3OMAK2GQJB7KBGAJHPHTL7GVWCCLCKDH2MK4DXWT6DQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.245067069 (17m28.026000444s) [noise=true , address=45.91.116.7:4011 , piece_id=2Q3VP4NNXV7FSIB3KTDKEI5VEA6FMJM3DV7WVXARUAAXIMVYPP5Q, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.245749945 (17m28.026683314s) [noise=true , address=45.91.116.8:4047 , piece_id=M6TIGG6K5IY5YLCWIIOAYM4OADHYT654KAHFZLAVHYRZFQXOMNSA, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.246361942 (17m28.02729532s) [noise=true , address=45.91.116.5:4299 , piece_id=MA22QRMEKYQGC4JHC73VHRZYUON6X5GSYCKPALUILHOTVE2JALZQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247224019 (17m28.028157391s) [noise=true , address=77.90.31.14:4663 , piece_id=4RJ57NHGYGNXNZ6WKJFSYTXABP5TCUW6SJ7N7BK6JYUI3CVXE6HQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247647098 (17m28.028580476s) [noise=true , address=45.91.116.10:4083 , piece_id=ULUVLWJPQ4GZASQJEH656LNH3LTDG437GOWBW2GCCVYALVSCLP2Q, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247771695 (17m28.028705064s) [noise=true , address=45.91.116.14:4190 , piece_id=YEFEQKDNF3RFW4YEUJG4CDDX2T4W5PYG2EBXZ6HOJTTGVD2L6LLQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.247883972 (17m28.028817342s) [noise=true , address=45.91.116.9:4262 , piece_id=7ROQYHMFIZSJE3E5IBV2YYRFBOKXZFPP4APP64TLO7EO2XTQFN3A, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.248440176 (17m28.029373547s) [noise=true , address=45.91.116.7:4010 , piece_id=4TGUR3523QK6DZ7ZNPLQIGMN5L6TM5XXQFDTIPUBVUSNMUALMTXQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:12.291635996 (17m28.07256936s) [noise=true , address=77.90.31.16:4734 , piece_id=KBQPSDTMMERJGT3L4L7WH536IO7LESN3RBNGGHVGG7UM5ABQKBQQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: <nil> Jul 26 22:09:42.031234530 (17m57.812167916s) [noise=true , address=104.36.82.8:4970 , piece_id=I6WEZKRVPH3HR4HHMETUYYJSMHWRWXN4MU5YAZHTY5KPGIFY3GKA, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031315006 (17m57.812248388s) [noise=true , address=104.36.82.15:4258 , piece_id=GWV7CGQUBC6KBHB5CPSUAKGEXZIRV554OMJLZCMV2FNCYDRPIHIQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 159.223.143.39:41696->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031326322 (17m57.812259705s) [noise=true , address=104.36.82.7:4911 , piece_id=GFYIHUPEHJCF75CAFM4CZ7KUK4XD445KOF7UBVDM4HXMHRPWZYZQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:42482->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031335779 (17m57.812269142s) [noise=true , address=104.36.82.6:4873 , piece_id=PC4W673ITXSO7A6OR5XR6J6K562JMQHJUNEDLWPWH3JJOLDDXALQ, seg_pos={0 0}, upload=276]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:09:42.031372559 (17m57.812305942s) [seg_pos={0 0}, upload=276]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:09:42.054859127 (17m57.835792503s) [upload=276]: Done uploading segments... Jul 26 22:09:42.054915101 (17m57.835848484s) [upload=276]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.026589943s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 723.802301ms FAIL (2025-07-26 22:09:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 723.826692ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 1m0.038405903s FAIL (2025-07-26 22:08:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== [noise=true , address=104.36.82.4:4813 , piece_id=U6V4VZNRGJ5EF3525Y5LDWNP6MCHJOL36E74CP3JL7GOVQIAYCZA, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.240257682 (16m4.021191066s) [noise=true , address=79.127.205.245:4249 , piece_id=Q4NGHX5W7NSLE3WPKE53QURMJMA5CR727BQKOZFGQZAXRD4PG2RQ, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.240548187 (16m4.021481574s) [noise=true , address=79.127.205.235:4682 , piece_id=2BDFVGEKEM5CGHK7HQYNGGCB2MK3JNX6UZUAJVAIMA4KNXMAYWPA, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.240654174 (16m4.021587559s) [noise=true , address=104.36.82.54:4179 , piece_id=KFJURZKXXYYQYBCVTKQUCAARSW3V7YF2A2KYO6UWZLQZDWYDVRFQ, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.241028922 (16m4.021962298s) [noise=true , address=104.36.82.9:4032 , piece_id=DTL5OAFKZ5A54ZKYX7MBHCHCMUEBCORB2HUDPL5VZA66QPXINRRQ, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.241079939 (16m4.022013324s) [noise=true , address=79.127.201.215:4351 , piece_id=GFODRC26IRJT5J2C2UVTKFM6OUN67OJXC2ICVWWKQB7K7LNKPWTA, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.241323796 (16m4.022257189s) [noise=true , address=79.127.219.38:4001 , piece_id=FHESJBHPNWRO2A7DJU7ZV4KN76QQKTSNLLJU6ZSOFCWGHKI4HK6Q, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.241619477 (16m4.02255286s) [noise=true , address=104.36.82.56:4238 , piece_id=56MCZHKDNLWEHOEXW4MWL3JE6XK5U55ZWBYFWPOY56CRMO3HHDEQ, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.241675349 (16m4.022608724s) [noise=true , address=156.146.43.225:4434 , piece_id=DDQ7RCGB3TVKOGEQQSEXDRWYTB2Y7HCABYIB64QF5J6S46KLE2EQ, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.241970785 (16m4.022904162s) [noise=true , address=104.36.82.55:4194 , piece_id=SI7N4OTQWZD6QWUVMS7DBZIQ5VOO5RXXP73FORMZBXKWVH6PNBBA, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.242108916 (16m4.0230423s) [noise=true , address=104.36.82.52:4108 , piece_id=2FBOCTOBO7FNTMJRIJLCTN6WGMXLNY4R3HAUJ277UYDEQORQTREQ, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.242307106 (16m4.023240482s) [noise=true , address=104.36.82.18:4391 , piece_id=VYXVTB7CTXWH4VOBIYC6EFGEBCOGPG4DZGAZCMKWBYEH27VAG3XA, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.242568332 (16m4.02350172s) [noise=true , address=104.36.82.7:4916 , piece_id=VLECULCMOLQPBMRHZBGLJYUY2NLLBCYSRI4RZRHD3L4SYFJXJSDA, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.242633989 (16m4.023567364s) [noise=true , address=104.36.82.6:4899 , piece_id=NMN5VG37RV2TQWUAESCAJQ5Q6TEE62WOLOLTGY34GBL4WCDQQQ7Q, seg_pos={0 0}, upload=254]: Uploading piece... Jul 26 22:07:48.319037558 (16m4.099970922s) [upload=254]: Next returned nil segment Jul 26 22:07:48.319054313 (16m4.09998768s) [upload=254]: Waiting for error group managing segments... Jul 26 22:07:48.375414232 (16m4.156347608s) [noise=true , address=77.90.31.13:4658 , piece_id=THZBQGBUCQCFGKQVOID3M4RIUDNN4EWTCHECRLZRZNPOORYM6WCA, seg_pos={0 0}, upload=254]: Done uploading piece. err: <nil> Jul 26 22:07:48.491833787 (16m4.272767159s) [noise=true , address=156.146.43.225:4434 , piece_id=DDQ7RCGB3TVKOGEQQSEXDRWYTB2Y7HCABYIB64QF5J6S46KLE2EQ, seg_pos={0 0}, upload=254]: Done uploading piece. err: <nil> Jul 26 22:07:48.521143633 (16m4.302077s) [noise=true , address=79.127.219.38:4001 , piece_id=FHESJBHPNWRO2A7DJU7ZV4KN76QQKTSNLLJU6ZSOFCWGHKI4HK6Q, seg_pos={0 0}, upload=254]: Done uploading piece. err: <nil> Jul 26 22:07:48.836434025 (16m4.617367389s) [noise=true , address=79.127.205.235:4682 , piece_id=2BDFVGEKEM5CGHK7HQYNGGCB2MK3JNX6UZUAJVAIMA4KNXMAYWPA, seg_pos={0 0}, upload=254]: Done uploading piece. err: <nil> Jul 26 22:07:48.855773612 (16m4.63670698s) [noise=true , address=79.127.205.245:4249 , piece_id=Q4NGHX5W7NSLE3WPKE53QURMJMA5CR727BQKOZFGQZAXRD4PG2RQ, seg_pos={0 0}, upload=254]: Done uploading piece. err: <nil> Jul 26 22:07:56.314289089 (16m12.095222458s) [noise=true , address=79.127.201.215:4351 , piece_id=GFODRC26IRJT5J2C2UVTKFM6OUN67OJXC2ICVWWKQB7K7LNKPWTA, seg_pos={0 0}, upload=254]: Done uploading piece. err: <nil> Jul 26 22:08:24.905663166 (16m40.686596531s) [noise=true , address=104.36.82.18:4391 , piece_id=VYXVTB7CTXWH4VOBIYC6EFGEBCOGPG4DZGAZCMKWBYEH27VAG3XA, seg_pos={0 0}, upload=254]: Done uploading piece. err: <nil> Jul 26 22:08:48.177123207 (17m3.958056593s) [noise=true , address=104.36.82.46:4481 , piece_id=CBTQM5BO5LPFH7Y2U5FL6K4PSFZPERPFBRMNR7A6S6K3JN5FIF5A, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 159.223.143.39:33896->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.177764198 (17m3.958697579s) [noise=true , address=104.36.82.47:4546 , piece_id=PULBBRBFA5FO2M53L4XB7EH7TJ2URQ7FJYE5DPQP2LDKNIL5CFIA, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: write tcp 159.223.143.39:43428->104.36.82.47:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.177839287 (17m3.958772665s) [noise=true , address=104.36.82.56:4238 , piece_id=56MCZHKDNLWEHOEXW4MWL3JE6XK5U55ZWBYFWPOY56CRMO3HHDEQ, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: write tcp 159.223.143.39:54868->104.36.82.56:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.177988660 (17m3.958922042s) [noise=true , address=104.36.82.9:4032 , piece_id=DTL5OAFKZ5A54ZKYX7MBHCHCMUEBCORB2HUDPL5VZA66QPXINRRQ, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ, address:104.36.82.9:4032): protocol: write tcp 159.223.143.39:51224->104.36.82.9:4032: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.178020829 (17m3.958954205s) [noise=true , address=104.36.82.54:4179 , piece_id=KFJURZKXXYYQYBCVTKQUCAARSW3V7YF2A2KYO6UWZLQZDWYDVRFQ, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP, address:104.36.82.54:4179): protocol: write tcp 159.223.143.39:47474->104.36.82.54:4179: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.178073811 (17m3.959007189s) [noise=true , address=104.36.82.9:4030 , piece_id=K6AJQXYEKHCQVVE3IIBU52VZHGRU3KLLQJM3NTRAMIWJF3YXDBSA, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:1MGCEjVE57ej5VuXgC6pYPFNUuAL851p2gqC53xxj6hEXvgYM5, address:104.36.82.9:4030): protocol: write tcp 159.223.143.39:39290->104.36.82.9:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.178110417 (17m3.959043793s) [noise=true , address=104.36.82.55:4194 , piece_id=SI7N4OTQWZD6QWUVMS7DBZIQ5VOO5RXXP73FORMZBXKWVH6PNBBA, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: write tcp 159.223.143.39:42472->104.36.82.55:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.178123271 (17m3.959056653s) [noise=true , address=104.36.82.52:4108 , piece_id=2FBOCTOBO7FNTMJRIJLCTN6WGMXLNY4R3HAUJ277UYDEQORQTREQ, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:12XJJZCUSHUS52YC6jhoPB4ayjsyKZT477PymPj4inkZtdUvheT, address:104.36.82.52:4108): protocol: write tcp 159.223.143.39:35076->104.36.82.52:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.178142542 (17m3.959075918s) [noise=true , address=104.36.82.6:4899 , piece_id=NMN5VG37RV2TQWUAESCAJQ5Q6TEE62WOLOLTGY34GBL4WCDQQQ7Q, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: write tcp 159.223.143.39:37214->104.36.82.6:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.178187909 (17m3.959121291s) [noise=true , address=104.36.82.4:4813 , piece_id=U6V4VZNRGJ5EF3525Y5LDWNP6MCHJOL36E74CP3JL7GOVQIAYCZA, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:104.36.82.4:4813): protocol: write tcp 159.223.143.39:46686->104.36.82.4:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.178225146 (17m3.959158526s) [noise=true , address=104.36.82.7:4916 , piece_id=VLECULCMOLQPBMRHZBGLJYUY2NLLBCYSRI4RZRHD3L4SYFJXJSDA, seg_pos={0 0}, upload=254]: Done uploading piece. err: ecclient: upload failed (node:12fifEPABGWmVzxBHNzhjh3r4ue3RRSeNdydifpXruXtH4FELBK, address:104.36.82.7:4916): protocol: write tcp 159.223.143.39:56956->104.36.82.7:4916: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:48.178259514 (17m3.959192892s) [seg_pos={0 0}, upload=254]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:08:48.207739085 (17m3.988672475s) [upload=254]: Done uploading segments... Jul 26 22:08:48.207773801 (17m3.988707181s) [upload=254]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.038431427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [......................................X..X........XXX.XXX.X.] duration: 536.856773ms FAIL (2025-07-26 22:08:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 536.884825ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......................................X..X........XXX.XXX.X] duration: 30.079599992s FAIL (2025-07-26 22:08:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.079696183s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......................................XX..XXX..X..XX..X..XX.] duration: 367.329365ms FAIL (2025-07-26 22:08:22 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== .82.13:4186 , piece_id=NXRK33672VD47XEOIR667ENXKBTEWWE3AY4AAGYMWDYH4LZBIQ7Q, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:60284->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857365078 (15m50.638298459s) [seg_pos={0 0}, upload=241]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 22:07:34.881995175 (15m50.662928543s) [upload=241]: Done uploading segments... Jul 26 22:07:34.882039661 (15m50.662973037s) [upload=241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 22:07:34.987498076 (15m50.768431441s) [upload=250]: Starting upload Jul 26 22:07:34.987564063 (15m50.768497438s) [upload=250]: Done starting upload Jul 26 22:07:34.987689434 (15m50.768622805s) [upload=250]: Uploading segments... Jul 26 22:07:34.987824291 (15m50.768757667s) [upload=250]: Got next segment. Inline: false Jul 26 22:07:34.987885176 (15m50.768818553s) [upload=250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:07:34.987891830 (15m50.768825202s) [upload=250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:07:35.016826140 (15m50.797759515s) [seg_pos={0 0}, upload=250]: Begin upload segment... Jul 26 22:07:35.016921473 (15m50.797854851s) [noise=true , address=104.36.82.14:4223 , piece_id=M5KS7VJAUIMN6WOOG3SNYJVRS3RZZFBBKXCSRPFXO2DHKWWO6ELQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.017468447 (15m50.798401832s) [noise=true , address=45.91.116.6:4973 , piece_id=SPG4LU2MFM52ECZWC3FBCGLMG6JREKJJLAJAOSEN3K7O5ZEATKMA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.017875249 (15m50.798808635s) [noise=true , address=77.90.31.10:4518 , piece_id=SBDHZMM4DIYCDSJGO7GFMABV5LNLHIQDGATTOIB2F5M36QP2TDHQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.018145431 (15m50.799078811s) [seg_pos={0 0}, upload=250]: Done begin upload segment. Jul 26 22:07:35.018172581 (15m50.799105965s) [noise=true , address=45.91.116.7:4011 , piece_id=NDIQKDW5CTHMLD5M4VRBNCV732TALHQYIZJOJ53MEINKCLSYS4SA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.018586538 (15m50.799519922s) [noise=true , address=104.36.82.5:4838 , piece_id=ENWNLJF2RCXPEOKPKUV6QT4CKMXIUNYSXLRJCJ7NF6F2D7AWCCCQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.018978715 (15m50.799912097s) [noise=true , address=45.91.116.14:4189 , piece_id=EI3JP5WRSE4DDCWDBENL77EIJZCCHR7U7KAW7PGRCOLJC7JBFRDA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.019607879 (15m50.800541265s) [noise=true , address=77.90.31.19:4843 , piece_id=DYS62SOG7O3F3LMM55CHBWR4GIFW3RYHGKOR6ATQTMXLABDQYGEQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.019859169 (15m50.800792548s) [noise=true , address=77.90.31.21:4878 , piece_id=GFWCU55RPIZWJI2W4Y4OON3GO6PMWYNHPTFT45YHK52E7VEN2YCA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.020584667 (15m50.801518035s) [noise=true , address=45.91.116.5:4297 , piece_id=7A4AXNUQW2GKI5F2YTH5OLSS5AEBCRR2U2QPJ6J37V73YAZFFG6Q, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.020634450 (15m50.801567817s) [noise=true , address=45.91.116.9:4263 , piece_id=4GH6GEETG6C3UTWPU72EJHADXHPHZG7WN4ID26BU7RIJJYUJ57OQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.021358305 (15m50.802291683s) [noise=true , address=45.91.116.13:4155 , piece_id=MKQYJLQWX5FW2TS7IGYARC6TNWLKXA7WAH7ZRPUCP65ZJAOLYXFQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.021726279 (15m50.802659658s) [noise=true , address=77.90.31.13:4626 , piece_id=7VP2RECC2FP6CMEMWUPCHUZBHJOHUIWPM5TL67L6ZKFDHR7KG7VQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.023351032 (15m50.804284442s) [noise=true , address=104.36.82.8:4971 , piece_id=OBX3R5SQTKY2LNYQFQFZVCSGBRZNSE3V6XJ7H6V3NUN5VOOSISFA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.023838014 (15m50.8047714s) [noise=true , address=45.91.116.9:4261 , piece_id=P22R47Q6SPYC4HRDJVMPMTDN74TTN225HMZ3GE76ZREIM4T5HUQQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.024344842 (15m50.805278227s) [noise=true , address=104.36.82.3:4765 , piece_id=JFWMJSBPHPYL42FLIMRHWSVXPSIBPW3F5EXMSL7LXNLFQDMWUO6A, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.024738414 (15m50.805671796s) [noise=true , address=45.91.116.8:4045 , piece_id=INJWCONIMF747RPLNSGL7WFNK4TUC7JCTK5ZWTLAYV5DVL7CDDPA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.112546258 (15m50.893479637s) [upload=250]: Next returned nil segment Jul 26 22:07:35.112579119 (15m50.8935125s) [upload=250]: Waiting for error group managing segments... Jul 26 22:07:35.115448922 (15m50.896382284s) [noise=true , address=45.91.116.7:4011 , piece_id=NDIQKDW5CTHMLD5M4VRBNCV732TALHQYIZJOJ53MEINKCLSYS4SA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.115834265 (15m50.896767627s) [noise=true , address=45.91.116.13:4155 , piece_id=MKQYJLQWX5FW2TS7IGYARC6TNWLKXA7WAH7ZRPUCP65ZJAOLYXFQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.116276376 (15m50.897209752s) [noise=true , address=45.91.116.6:4973 , piece_id=SPG4LU2MFM52ECZWC3FBCGLMG6JREKJJLAJAOSEN3K7O5ZEATKMA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.116721260 (15m50.897654636s) [noise=true , address=45.91.116.5:4297 , piece_id=7A4AXNUQW2GKI5F2YTH5OLSS5AEBCRR2U2QPJ6J37V73YAZFFG6Q, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.117075039 (15m50.898008407s) [noise=true , address=45.91.116.14:4189 , piece_id=EI3JP5WRSE4DDCWDBENL77EIJZCCHR7U7KAW7PGRCOLJC7JBFRDA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.117447964 (15m50.898381334s) [noise=true , address=45.91.116.9:4261 , piece_id=P22R47Q6SPYC4HRDJVMPMTDN74TTN225HMZ3GE76ZREIM4T5HUQQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.117922945 (15m50.898856313s) [noise=true , address=45.91.116.8:4045 , piece_id=INJWCONIMF747RPLNSGL7WFNK4TUC7JCTK5ZWTLAYV5DVL7CDDPA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.118400510 (15m50.899333892s) [noise=true , address=77.90.31.19:4843 , piece_id=DYS62SOG7O3F3LMM55CHBWR4GIFW3RYHGKOR6ATQTMXLABDQYGEQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.118978228 (15m50.899911603s) [noise=true , address=77.90.31.21:4878 , piece_id=GFWCU55RPIZWJI2W4Y4OON3GO6PMWYNHPTFT45YHK52E7VEN2YCA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.124493998 (15m50.905427362s) [noise=true , address=45.91.116.9:4263 , piece_id=4GH6GEETG6C3UTWPU72EJHADXHPHZG7WN4ID26BU7RIJJYUJ57OQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.569833987 (15m51.350767353s) [noise=true , address=77.90.31.13:4626 , piece_id=7VP2RECC2FP6CMEMWUPCHUZBHJOHUIWPM5TL67L6ZKFDHR7KG7VQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:08:04.957316577 (16m20.738249955s) [noise=true , address=77.90.31.10:4518 , piece_id=SBDHZMM4DIYCDSJGO7GFMABV5LNLHIQDGATTOIB2F5M36QP2TDHQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 159.223.143.39:42898->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957422534 (16m20.73835593s) [noise=true , address=104.36.82.14:4223 , piece_id=M5KS7VJAUIMN6WOOG3SNYJVRS3RZZFBBKXCSRPFXO2DHKWWO6ELQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:33464->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957456634 (16m20.738390015s) [noise=true , address=104.36.82.3:4765 , piece_id=JFWMJSBPHPYL42FLIMRHWSVXPSIBPW3F5EXMSL7LXNLFQDMWUO6A, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:60924->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957485814 (16m20.738419177s) [noise=true , address=104.36.82.5:4838 , piece_id=ENWNLJF2RCXPEOKPKUV6QT4CKMXIUNYSXLRJCJ7NF6F2D7AWCCCQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:43946->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957499116 (16m20.738432493s) [noise=true , address=104.36.82.8:4971 , piece_id=OBX3R5SQTKY2LNYQFQFZVCSGBRZNSE3V6XJ7H6V3NUN5VOOSISFA, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:40640->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957544370 (16m20.738477756s) [seg_pos={0 0}, upload=250]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:08:04.979268646 (16m20.760202015s) [upload=250]: Done uploading segments... Jul 26 22:08:04.979309377 (16m20.760242755s) [upload=250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 367.358918ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 653.457644ms FAIL (2025-07-26 22:08:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 653.47974ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......................................XX..XXX..X..XX..X..XX] duration: 30.024568075s FAIL (2025-07-26 22:08:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== .82.13:4186 , piece_id=NXRK33672VD47XEOIR667ENXKBTEWWE3AY4AAGYMWDYH4LZBIQ7Q, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:60284->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857365078 (15m50.638298459s) [seg_pos={0 0}, upload=241]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 22:07:34.881995175 (15m50.662928543s) [upload=241]: Done uploading segments... Jul 26 22:07:34.882039661 (15m50.662973037s) [upload=241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 22:07:34.987498076 (15m50.768431441s) [upload=250]: Starting upload Jul 26 22:07:34.987564063 (15m50.768497438s) [upload=250]: Done starting upload Jul 26 22:07:34.987689434 (15m50.768622805s) [upload=250]: Uploading segments... Jul 26 22:07:34.987824291 (15m50.768757667s) [upload=250]: Got next segment. Inline: false Jul 26 22:07:34.987885176 (15m50.768818553s) [upload=250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:07:34.987891830 (15m50.768825202s) [upload=250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:07:35.016826140 (15m50.797759515s) [seg_pos={0 0}, upload=250]: Begin upload segment... Jul 26 22:07:35.016921473 (15m50.797854851s) [noise=true , address=104.36.82.14:4223 , piece_id=M5KS7VJAUIMN6WOOG3SNYJVRS3RZZFBBKXCSRPFXO2DHKWWO6ELQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.017468447 (15m50.798401832s) [noise=true , address=45.91.116.6:4973 , piece_id=SPG4LU2MFM52ECZWC3FBCGLMG6JREKJJLAJAOSEN3K7O5ZEATKMA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.017875249 (15m50.798808635s) [noise=true , address=77.90.31.10:4518 , piece_id=SBDHZMM4DIYCDSJGO7GFMABV5LNLHIQDGATTOIB2F5M36QP2TDHQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.018145431 (15m50.799078811s) [seg_pos={0 0}, upload=250]: Done begin upload segment. Jul 26 22:07:35.018172581 (15m50.799105965s) [noise=true , address=45.91.116.7:4011 , piece_id=NDIQKDW5CTHMLD5M4VRBNCV732TALHQYIZJOJ53MEINKCLSYS4SA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.018586538 (15m50.799519922s) [noise=true , address=104.36.82.5:4838 , piece_id=ENWNLJF2RCXPEOKPKUV6QT4CKMXIUNYSXLRJCJ7NF6F2D7AWCCCQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.018978715 (15m50.799912097s) [noise=true , address=45.91.116.14:4189 , piece_id=EI3JP5WRSE4DDCWDBENL77EIJZCCHR7U7KAW7PGRCOLJC7JBFRDA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.019607879 (15m50.800541265s) [noise=true , address=77.90.31.19:4843 , piece_id=DYS62SOG7O3F3LMM55CHBWR4GIFW3RYHGKOR6ATQTMXLABDQYGEQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.019859169 (15m50.800792548s) [noise=true , address=77.90.31.21:4878 , piece_id=GFWCU55RPIZWJI2W4Y4OON3GO6PMWYNHPTFT45YHK52E7VEN2YCA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.020584667 (15m50.801518035s) [noise=true , address=45.91.116.5:4297 , piece_id=7A4AXNUQW2GKI5F2YTH5OLSS5AEBCRR2U2QPJ6J37V73YAZFFG6Q, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.020634450 (15m50.801567817s) [noise=true , address=45.91.116.9:4263 , piece_id=4GH6GEETG6C3UTWPU72EJHADXHPHZG7WN4ID26BU7RIJJYUJ57OQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.021358305 (15m50.802291683s) [noise=true , address=45.91.116.13:4155 , piece_id=MKQYJLQWX5FW2TS7IGYARC6TNWLKXA7WAH7ZRPUCP65ZJAOLYXFQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.021726279 (15m50.802659658s) [noise=true , address=77.90.31.13:4626 , piece_id=7VP2RECC2FP6CMEMWUPCHUZBHJOHUIWPM5TL67L6ZKFDHR7KG7VQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.023351032 (15m50.804284442s) [noise=true , address=104.36.82.8:4971 , piece_id=OBX3R5SQTKY2LNYQFQFZVCSGBRZNSE3V6XJ7H6V3NUN5VOOSISFA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.023838014 (15m50.8047714s) [noise=true , address=45.91.116.9:4261 , piece_id=P22R47Q6SPYC4HRDJVMPMTDN74TTN225HMZ3GE76ZREIM4T5HUQQ, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.024344842 (15m50.805278227s) [noise=true , address=104.36.82.3:4765 , piece_id=JFWMJSBPHPYL42FLIMRHWSVXPSIBPW3F5EXMSL7LXNLFQDMWUO6A, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.024738414 (15m50.805671796s) [noise=true , address=45.91.116.8:4045 , piece_id=INJWCONIMF747RPLNSGL7WFNK4TUC7JCTK5ZWTLAYV5DVL7CDDPA, seg_pos={0 0}, upload=250]: Uploading piece... Jul 26 22:07:35.112546258 (15m50.893479637s) [upload=250]: Next returned nil segment Jul 26 22:07:35.112579119 (15m50.8935125s) [upload=250]: Waiting for error group managing segments... Jul 26 22:07:35.115448922 (15m50.896382284s) [noise=true , address=45.91.116.7:4011 , piece_id=NDIQKDW5CTHMLD5M4VRBNCV732TALHQYIZJOJ53MEINKCLSYS4SA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.115834265 (15m50.896767627s) [noise=true , address=45.91.116.13:4155 , piece_id=MKQYJLQWX5FW2TS7IGYARC6TNWLKXA7WAH7ZRPUCP65ZJAOLYXFQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.116276376 (15m50.897209752s) [noise=true , address=45.91.116.6:4973 , piece_id=SPG4LU2MFM52ECZWC3FBCGLMG6JREKJJLAJAOSEN3K7O5ZEATKMA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.116721260 (15m50.897654636s) [noise=true , address=45.91.116.5:4297 , piece_id=7A4AXNUQW2GKI5F2YTH5OLSS5AEBCRR2U2QPJ6J37V73YAZFFG6Q, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.117075039 (15m50.898008407s) [noise=true , address=45.91.116.14:4189 , piece_id=EI3JP5WRSE4DDCWDBENL77EIJZCCHR7U7KAW7PGRCOLJC7JBFRDA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.117447964 (15m50.898381334s) [noise=true , address=45.91.116.9:4261 , piece_id=P22R47Q6SPYC4HRDJVMPMTDN74TTN225HMZ3GE76ZREIM4T5HUQQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.117922945 (15m50.898856313s) [noise=true , address=45.91.116.8:4045 , piece_id=INJWCONIMF747RPLNSGL7WFNK4TUC7JCTK5ZWTLAYV5DVL7CDDPA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.118400510 (15m50.899333892s) [noise=true , address=77.90.31.19:4843 , piece_id=DYS62SOG7O3F3LMM55CHBWR4GIFW3RYHGKOR6ATQTMXLABDQYGEQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.118978228 (15m50.899911603s) [noise=true , address=77.90.31.21:4878 , piece_id=GFWCU55RPIZWJI2W4Y4OON3GO6PMWYNHPTFT45YHK52E7VEN2YCA, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.124493998 (15m50.905427362s) [noise=true , address=45.91.116.9:4263 , piece_id=4GH6GEETG6C3UTWPU72EJHADXHPHZG7WN4ID26BU7RIJJYUJ57OQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:07:35.569833987 (15m51.350767353s) [noise=true , address=77.90.31.13:4626 , piece_id=7VP2RECC2FP6CMEMWUPCHUZBHJOHUIWPM5TL67L6ZKFDHR7KG7VQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: <nil> Jul 26 22:08:04.957316577 (16m20.738249955s) [noise=true , address=77.90.31.10:4518 , piece_id=SBDHZMM4DIYCDSJGO7GFMABV5LNLHIQDGATTOIB2F5M36QP2TDHQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 159.223.143.39:42898->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957422534 (16m20.73835593s) [noise=true , address=104.36.82.14:4223 , piece_id=M5KS7VJAUIMN6WOOG3SNYJVRS3RZZFBBKXCSRPFXO2DHKWWO6ELQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:33464->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957456634 (16m20.738390015s) [noise=true , address=104.36.82.3:4765 , piece_id=JFWMJSBPHPYL42FLIMRHWSVXPSIBPW3F5EXMSL7LXNLFQDMWUO6A, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:60924->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957485814 (16m20.738419177s) [noise=true , address=104.36.82.5:4838 , piece_id=ENWNLJF2RCXPEOKPKUV6QT4CKMXIUNYSXLRJCJ7NF6F2D7AWCCCQ, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:43946->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957499116 (16m20.738432493s) [noise=true , address=104.36.82.8:4971 , piece_id=OBX3R5SQTKY2LNYQFQFZVCSGBRZNSE3V6XJ7H6V3NUN5VOOSISFA, seg_pos={0 0}, upload=250]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:40640->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:08:04.957544370 (16m20.738477756s) [seg_pos={0 0}, upload=250]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:08:04.979268646 (16m20.760202015s) [upload=250]: Done uploading segments... Jul 26 22:08:04.979309377 (16m20.760242755s) [upload=250]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.024590963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................................X..X........XXX.XXX.] duration: 588.009088ms FAIL (2025-07-26 22:07:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 588.036895ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 565.482094ms FAIL (2025-07-26 22:07:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 565.516576ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.041606098s FAIL (2025-07-26 22:07:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== , piece_id=BCZOBDMVEVOQAOGUS566GMMWHO4G2YRGPKVFVJLLNN3UWK644Q5Q, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877006092 (14m56.65793947s) [noise=true , address=104.36.82.41:4783 , piece_id=EAYWWTXHBEUQWK2MTOASIZJ7WUZYHHO3NNJA72OLUJ357VWTRV3Q, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877310238 (14m56.658243613s) [noise=true , address=104.36.82.52:4092 , piece_id=Y6JU6PBUAHAOWHFYPEN2Q3OTY6YEM4QPMDLK3WXFY6QLSE4GHEMA, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877379938 (14m56.658313324s) [noise=true , address=104.36.82.46:4508 , piece_id=Q6CPIWSJS67CFDO3BQINFHIATZHAE5UA632KP3BDML6QVMDQAFXQ, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877637090 (14m56.65857047s) [noise=true , address=77.90.31.21:4882 , piece_id=JAODWQCKYYGIGJTWLBFLZ54SARW3RPM34OWHC2VBUHCL24OIHZWQ, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877819176 (14m56.658752541s) [noise=true , address=104.36.82.53:4150 , piece_id=ZEEGY53KJBANKX75IL3IHXSGGWT4VNPWPK6WQAIFZMM76GP3IZ7A, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.877948680 (14m56.658882057s) [noise=true , address=79.127.201.215:4358 , piece_id=F5FPDFY525VHMUREAJWTMVWM4GEH4LUCWVVJQHF3M766AAORYMVQ, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.878274977 (14m56.659208357s) [noise=true , address=104.36.82.15:4283 , piece_id=463BFK2XQGX4RYFKUUZWA47NB4QTPIY3XU2E2MAO5KYBWCRP6IFQ, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.878596455 (14m56.659529834s) [noise=true , address=104.36.82.35:4602 , piece_id=VTQQJNCGPO6XIZGHVTHAKREY7M7D55URTJX4JAKDXXHLUHWZU26A, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.879263348 (14m56.660196733s) [noise=true , address=104.36.82.56:4261 , piece_id=EFIA3MY2I7H2HFXUMW4YCPV65SV5IJ6OG2GJEUW742G2N4W243QQ, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.879823357 (14m56.660756719s) [noise=true , address=104.36.82.10:4098 , piece_id=CXVIW4YYSBS3LARQVQ3AXWBVRCRZ7X7LP5GZSR3KIC2OOS4POZXA, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.880462649 (14m56.661396034s) [noise=true , address=64.49.13.155:4916 , piece_id=2FKOFXTHMCXDQVGEJ3MM7U72VLSSMYW6FRDBJE3F2XZCKYFPS3ZA, seg_pos={0 0}, upload=234]: Uploading piece... Jul 26 22:06:40.942586552 (14m56.723519942s) [upload=234]: Next returned nil segment Jul 26 22:06:40.942607586 (14m56.723540972s) [upload=234]: Waiting for error group managing segments... Jul 26 22:06:40.951178134 (14m56.732111514s) [noise=true , address=77.90.31.21:4882 , piece_id=JAODWQCKYYGIGJTWLBFLZ54SARW3RPM34OWHC2VBUHCL24OIHZWQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: <nil> Jul 26 22:06:41.480824114 (14m57.261757476s) [noise=true , address=79.127.205.234:4613 , piece_id=25SSVYOEJI2PVKEB4NTP2PILDOWUAMZL7I2QR7UFNKYLEU3JIMHA, seg_pos={0 0}, upload=234]: Done uploading piece. err: <nil> Jul 26 22:06:41.566016795 (14m57.346950175s) [noise=true , address=64.49.13.155:4916 , piece_id=2FKOFXTHMCXDQVGEJ3MM7U72VLSSMYW6FRDBJE3F2XZCKYFPS3ZA, seg_pos={0 0}, upload=234]: Done uploading piece. err: <nil> Jul 26 22:06:41.865601601 (14m57.646534984s) [noise=true , address=156.146.43.225:4448 , piece_id=ILJ54PWU4YEYQ6DIBEKPJUEY3RS6EPBKXFLYGFCLESRYBFBPJKCQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:06:43.354758170 (14m59.135691615s) [noise=true , address=79.127.201.215:4358 , piece_id=F5FPDFY525VHMUREAJWTMVWM4GEH4LUCWVVJQHF3M766AAORYMVQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: <nil> Jul 26 22:07:40.800116216 (15m56.581049577s) [noise=true , address=104.36.82.53:4150 , piece_id=ZEEGY53KJBANKX75IL3IHXSGGWT4VNPWPK6WQAIFZMM76GP3IZ7A, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: rpc: dial tcp 104.36.82.53:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800316561 (15m56.581249949s) [noise=true , address=104.36.82.56:4261 , piece_id=EFIA3MY2I7H2HFXUMW4YCPV65SV5IJ6OG2GJEUW742G2N4W243QQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: write tcp 159.223.143.39:38844->104.36.82.56:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800383993 (15m56.581317369s) [noise=true , address=104.36.82.10:4098 , piece_id=CXVIW4YYSBS3LARQVQ3AXWBVRCRZ7X7LP5GZSR3KIC2OOS4POZXA, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: rpc: dial tcp 104.36.82.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800464580 (15m56.581397956s) [noise=true , address=104.36.82.35:4602 , piece_id=VTQQJNCGPO6XIZGHVTHAKREY7M7D55URTJX4JAKDXXHLUHWZU26A, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:12oPZgRG4QqgnQLo1fgyGYoX1zd34wsd5n7p7L3mRbPD8B4JUuB, address:104.36.82.35:4602): protocol: write tcp 159.223.143.39:47488->104.36.82.35:4602: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800520570 (15m56.581453946s) [noise=true , address=104.36.82.36:4640 , piece_id=K7HNZOSXETXS2Y77ICWIXX2BG37L4Y6QRZKJIJLGRHCZ6RKT3XHQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1cL33G9zW6AsBkSfbrYC5DH1hk3ffBoNnt6CoeEy1D8SKS2hhS, address:104.36.82.36:4640): protocol: write tcp 159.223.143.39:54832->104.36.82.36:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800559320 (15m56.581492695s) [noise=true , address=104.36.82.47:4525 , piece_id=LCXBDWBOFL7GTBNVUZA6K7TREVIR6INMVXN5XYZQQGPN67TBLM5Q, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: write tcp 159.223.143.39:37796->104.36.82.47:4525: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800599019 (15m56.58153238s) [noise=true , address=104.36.82.41:4783 , piece_id=EAYWWTXHBEUQWK2MTOASIZJ7WUZYHHO3NNJA72OLUJ357VWTRV3Q, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz, address:104.36.82.41:4783): protocol: write tcp 159.223.143.39:36450->104.36.82.41:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800636207 (15m56.581569582s) [noise=true , address=104.36.82.52:4092 , piece_id=Y6JU6PBUAHAOWHFYPEN2Q3OTY6YEM4QPMDLK3WXFY6QLSE4GHEMA, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 159.223.143.39:43048->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800676301 (15m56.581609676s) [noise=true , address=104.36.82.46:4508 , piece_id=Q6CPIWSJS67CFDO3BQINFHIATZHAE5UA632KP3BDML6QVMDQAFXQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: rpc: dial tcp 104.36.82.46:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800719140 (15m56.581652501s) [noise=true , address=104.36.82.15:4283 , piece_id=463BFK2XQGX4RYFKUUZWA47NB4QTPIY3XU2E2MAO5KYBWCRP6IFQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU, address:104.36.82.15:4283): protocol: write tcp 159.223.143.39:34268->104.36.82.15:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800739495 (15m56.581672857s) [noise=true , address=104.36.82.40:4950 , piece_id=S52SGC3KQSNK7GDRBVYU3YKXKF6QFKWYF6CGT4IYGHATBHSMH33Q, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk, address:104.36.82.40:4950): protocol: rpc: dial tcp 104.36.82.40:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800757982 (15m56.581691357s) [noise=true , address=104.36.82.54:4189 , piece_id=ZB6LBJCD2OBUW63NIQ3JDFOTGHTOWQBWMW2VCECEXIVM26HWJYYQ, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: rpc: dial tcp 104.36.82.54:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:40.800895980 (15m56.581829358s) [noise=true , address=104.36.82.38:4707 , piece_id=BCZOBDMVEVOQAOGUS566GMMWHO4G2YRGPKVFVJLLNN3UWK644Q5Q, seg_pos={0 0}, upload=234]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:104.36.82.38:4707): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:07:40.800911905 (15m56.581845281s) [seg_pos={0 0}, upload=234]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 22:07:40.839274557 (15m56.620207933s) [upload=234]: Done uploading segments... Jul 26 22:07:40.839304866 (15m56.620238243s) [upload=234]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.041622514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................................XX..XXX..X..XX..X..X] duration: 30.027344199s FAIL (2025-07-26 22:07:34 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ce. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: context canceled; context canceled Jul 26 22:06:31.233378398 (14m47.014311764s) [seg_pos={0 0}, upload=232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:06:31.233469029 (14m47.014402405s) [upload=232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:06:31.233476080 (14m47.014409455s) [upload=232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:06:31.278182361 (14m47.059115727s) [upload=232]: Done uploading segments... Jul 26 22:06:31.278200919 (14m47.059134284s) [upload=232]: Upload finished. err: <nil> Jul 26 22:07:04.886004988 (15m20.666938357s) [upload=241]: Starting upload Jul 26 22:07:04.886083010 (15m20.667016377s) [upload=241]: Done starting upload Jul 26 22:07:04.886216997 (15m20.667150366s) [upload=241]: Uploading segments... Jul 26 22:07:04.886399538 (15m20.667332914s) [upload=241]: Got next segment. Inline: false Jul 26 22:07:04.886437605 (15m20.66737098s) [upload=241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:07:04.886442975 (15m20.66737635s) [upload=241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:07:04.913653381 (15m20.694586766s) [seg_pos={0 0}, upload=241]: Begin upload segment... Jul 26 22:07:04.913704315 (15m20.694637701s) [seg_pos={0 0}, upload=241]: Done begin upload segment. Jul 26 22:07:04.913734987 (15m20.694668373s) [noise=true , address=77.90.31.16:4735 , piece_id=ABL77ZP3AYR55JMLZSJQPNS5MADKSVXLRG4HNNHCQ3RI6XJVSN3A, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.914292152 (15m20.69522554s) [noise=true , address=104.36.82.6:4873 , piece_id=RMOJ6XAZROKNZ5UXQNHZQSK72CLQDF4YZC7YPF6XO5LQBHKSLWYA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.914734764 (15m20.69566815s) [noise=true , address=45.91.116.7:4011 , piece_id=32PGHV3RKESO3I7NHKAT6FPBLUPA2YXX7YBMIVDRZB3T3CZMQSIQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.915201856 (15m20.696135238s) [noise=true , address=104.36.82.8:4971 , piece_id=JTAWVK5TNVLYXEAH3I76YS6LEMBWB5VXXZIVSWVJCP2MTK62CNBQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.915643883 (15m20.69657727s) [noise=true , address=45.91.116.11:4117 , piece_id=S4C6HHEXO6THUQKOT4KDCCUAXZJROY4O4E5TBBBGUY4G5CL35ERA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.916071795 (15m20.697005181s) [noise=true , address=45.91.116.12:4225 , piece_id=HO33CJOKMCLPHCQBMBN55GYORDG2LBZKJSZU75FP7VO7DJWLO74Q, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.916615660 (15m20.697549043s) [noise=true , address=45.91.116.12:4226 , piece_id=7JZIBEFPYFEZHAJJBTJ6HZWJKCQ7HJ4RNTAYZ3HSWOM2NSWAC6QA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.917030219 (15m20.697963595s) [noise=true , address=45.91.116.10:4082 , piece_id=KSATKOX4SOOPYQNC6VE3QS3OZODW3HVQAJJMA2UIXIOQXSZI7YAQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.917308642 (15m20.698242017s) [noise=true , address=77.90.31.19:4841 , piece_id=5HRMIX7NQ7TCRXMYXITOUXTYM242KKI4IC2LJXWHL4ZYI5EMRUVA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.917581133 (15m20.698514508s) [noise=true , address=45.91.116.5:4297 , piece_id=PAQ47YOYSS7A4WZVWFVUVZJQ3VROOJNTYKLATHIUC2X4UKTA4FGQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.917845509 (15m20.698778884s) [noise=true , address=45.91.116.14:4191 , piece_id=MUN6GXT527TPRWO7R4STIPA4W7F2JA5CQGYZ4MB6Z5IMMPZ4E7WA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.918196794 (15m20.699130178s) [noise=true , address=104.36.82.3:4765 , piece_id=MPM2QMWXZAXYLOBESY4A5QKMSVMABKVEJQLT3ZY7FHBN6JD5OW6Q, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.918601274 (15m20.699534659s) [noise=true , address=45.91.116.13:4153 , piece_id=RXYGO2GR3Y7EWO2Z7CVFWLXGUVTPRVBM7YNIRFUCVLGOLYHSRZRA, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.921774832 (15m20.702708217s) [noise=true , address=104.36.82.13:4186 , piece_id=NXRK33672VD47XEOIR667ENXKBTEWWE3AY4AAGYMWDYH4LZBIQ7Q, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.922268298 (15m20.703201683s) [noise=true , address=77.90.31.10:4517 , piece_id=QEPL5S5FEYYPUTKHWJYZPSGIQXSD2J3ISA64KGDBS5HYEYJRWZKQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:04.922742454 (15m20.70367584s) [noise=true , address=104.36.82.7:4910 , piece_id=QCAXQDKO44Q22NJ5SCEX44Q2QEUD4CQDPOWV6MHAIWM4PI4F6SAQ, seg_pos={0 0}, upload=241]: Uploading piece... Jul 26 22:07:05.008625277 (15m20.789558673s) [upload=241]: Next returned nil segment Jul 26 22:07:05.008658994 (15m20.789592389s) [upload=241]: Waiting for error group managing segments... Jul 26 22:07:05.012782416 (15m20.7937158s) [noise=true , address=45.91.116.12:4226 , piece_id=7JZIBEFPYFEZHAJJBTJ6HZWJKCQ7HJ4RNTAYZ3HSWOM2NSWAC6QA, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.013480505 (15m20.794413887s) [noise=true , address=45.91.116.5:4297 , piece_id=PAQ47YOYSS7A4WZVWFVUVZJQ3VROOJNTYKLATHIUC2X4UKTA4FGQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.013559337 (15m20.794492712s) [noise=true , address=45.91.116.7:4011 , piece_id=32PGHV3RKESO3I7NHKAT6FPBLUPA2YXX7YBMIVDRZB3T3CZMQSIQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.013876421 (15m20.794809783s) [noise=true , address=45.91.116.13:4153 , piece_id=RXYGO2GR3Y7EWO2Z7CVFWLXGUVTPRVBM7YNIRFUCVLGOLYHSRZRA, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.014266643 (15m20.795200008s) [noise=true , address=45.91.116.14:4191 , piece_id=MUN6GXT527TPRWO7R4STIPA4W7F2JA5CQGYZ4MB6Z5IMMPZ4E7WA, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.014367475 (15m20.795300845s) [noise=true , address=45.91.116.12:4225 , piece_id=HO33CJOKMCLPHCQBMBN55GYORDG2LBZKJSZU75FP7VO7DJWLO74Q, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.015368047 (15m20.796301413s) [noise=true , address=45.91.116.11:4117 , piece_id=S4C6HHEXO6THUQKOT4KDCCUAXZJROY4O4E5TBBBGUY4G5CL35ERA, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.016610390 (15m20.79754376s) [noise=true , address=77.90.31.19:4841 , piece_id=5HRMIX7NQ7TCRXMYXITOUXTYM242KKI4IC2LJXWHL4ZYI5EMRUVA, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.023964727 (15m20.804898094s) [noise=true , address=45.91.116.10:4082 , piece_id=KSATKOX4SOOPYQNC6VE3QS3OZODW3HVQAJJMA2UIXIOQXSZI7YAQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:05.768989621 (15m21.54992299s) [noise=true , address=77.90.31.16:4735 , piece_id=ABL77ZP3AYR55JMLZSJQPNS5MADKSVXLRG4HNNHCQ3RI6XJVSN3A, seg_pos={0 0}, upload=241]: Done uploading piece. err: <nil> Jul 26 22:07:34.857092253 (15m50.638025619s) [noise=true , address=104.36.82.7:4910 , piece_id=QCAXQDKO44Q22NJ5SCEX44Q2QEUD4CQDPOWV6MHAIWM4PI4F6SAQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857153354 (15m50.638086721s) [noise=true , address=104.36.82.6:4873 , piece_id=RMOJ6XAZROKNZ5UXQNHZQSK72CLQDF4YZC7YPF6XO5LQBHKSLWYA, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857217544 (15m50.638150909s) [noise=true , address=104.36.82.3:4765 , piece_id=MPM2QMWXZAXYLOBESY4A5QKMSVMABKVEJQLT3ZY7FHBN6JD5OW6Q, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857273731 (15m50.638207106s) [noise=true , address=77.90.31.10:4517 , piece_id=QEPL5S5FEYYPUTKHWJYZPSGIQXSD2J3ISA64KGDBS5HYEYJRWZKQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857280934 (15m50.638214302s) [noise=true , address=104.36.82.8:4971 , piece_id=JTAWVK5TNVLYXEAH3I76YS6LEMBWB5VXXZIVSWVJCP2MTK62CNBQ, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857334028 (15m50.638267392s) [noise=true , address=104.36.82.13:4186 , piece_id=NXRK33672VD47XEOIR667ENXKBTEWWE3AY4AAGYMWDYH4LZBIQ7Q, seg_pos={0 0}, upload=241]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:60284->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:07:34.857365078 (15m50.638298459s) [seg_pos={0 0}, upload=241]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 22:07:34.881995175 (15m50.662928543s) [upload=241]: Done uploading segments... Jul 26 22:07:34.882039661 (15m50.662973037s) [upload=241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.027385728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................................X..X........XXX.XXX] duration: 30.038421607s FAIL (2025-07-26 22:07:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.038442419s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.08825315s FAIL (2025-07-26 22:07:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.088574587s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................................X..X........XXX.XX] duration: 30.033474142s FAIL (2025-07-26 22:06:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.033498718s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)