Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..X...........X.X....X.XX.....................X........] duration: 550.017743ms FAIL (2025-07-28 01:44:29 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 550.051309ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X............X........................................] duration: 224.531856ms FAIL (2025-07-28 01:44:06 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 224.603984ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................X..XX.X..X.X...................] duration: 424.646192ms FAIL (2025-07-28 01:43:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 424.680872ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..................] duration: 13.261119093s FAIL (2025-07-28 01:43:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 13.26117722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X..X...........X.X....X.XX.....................X.......] duration: 506.078016ms FAIL (2025-07-28 01:43:44 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 506.113533ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X............X.......................................] duration: 203.501071ms FAIL (2025-07-28 01:43:44 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 203.528778ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X............X......................................] duration: 289.924296ms FAIL (2025-07-28 01:43:23 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 289.956258ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................X..XX.X..X.X..................] duration: 499.835865ms FAIL (2025-07-28 01:43:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 499.870466ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X..X...........X.X....X.XX.....................X......] duration: 448.654206ms FAIL (2025-07-28 01:42:44 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 448.688458ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.................] duration: 8.976004198s FAIL (2025-07-28 01:42:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 8.976038699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................X..XX.X..X.X.................] duration: 533.092658ms FAIL (2025-07-28 01:42:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 533.138168ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X............X.....................................] duration: 257.564397ms FAIL (2025-07-28 01:42:10 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 257.665154ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................] duration: 22.720214434s FAIL (2025-07-28 01:41:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 22.720273549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X..X...........X.X....X.XX.....................X.....] duration: 3.17739232s FAIL (2025-07-28 01:41:43 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 3.177418803s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................X..XX.X..X.X................] duration: 410.428744ms FAIL (2025-07-28 01:41:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.523469ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X............X....................................] duration: 192.463994ms FAIL (2025-07-28 01:41:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 192.49997ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X..X...........X.X....X.XX.....................X....] duration: 457.880656ms FAIL (2025-07-28 01:41:04 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 457.911751ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...............] duration: 11.345592701s FAIL (2025-07-28 01:40:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 11.345622978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................X..XX.X..X.X...............] duration: 416.935142ms FAIL (2025-07-28 01:40:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 416.971986ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............] duration: 3.416999755s FAIL (2025-07-28 01:40:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.41703357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X............X...................................] duration: 199.077213ms FAIL (2025-07-28 01:40:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 199.166501ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [X........X..X...........X.X....X.XX.....................X...] duration: 487.379432ms FAIL (2025-07-28 01:39:58 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 487.40828ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................X..XX.X..X.X..............] duration: 430.896973ms FAIL (2025-07-28 01:39:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 430.932239ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X............X..................................] duration: 218.063265ms FAIL (2025-07-28 01:39:31 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 218.096862ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX........X..X...........X.X....X.XX.....................X..] duration: 555.877512ms FAIL (2025-07-28 01:39:26 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 555.915662ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.............] duration: 3.840943075s FAIL (2025-07-28 01:39:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.840973174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X............X.................................] duration: 445.229993ms FAIL (2025-07-28 01:39:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 445.279453ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................X..XX.X..X.X.............] duration: 535.547968ms FAIL (2025-07-28 01:39:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 535.583296ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX............] duration: 13.428919563s FAIL (2025-07-28 01:38:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 13.428954059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X............X................................] duration: 273.441956ms FAIL (2025-07-28 01:38:29 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 273.477447ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX........X..X...........X.X....X.XX.....................X.] duration: 424.036971ms FAIL (2025-07-28 01:38:20 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 424.066841ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration: 13.735653481s FAIL (2025-07-28 01:38:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 13.735691565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................X..XX.X..X.X............] duration: 690.222328ms FAIL (2025-07-28 01:38:03 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 690.261229ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [..XX........X..X...........X.X....X.XX.....................X] duration: 323.297019ms FAIL (2025-07-28 01:37:41 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 323.402667ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X............X...............................] duration: 248.649888ms FAIL (2025-07-28 01:37:41 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 248.854997ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [...XX........X..X...........X.X....X.XX.....................] duration: 646.864722ms FAIL (2025-07-28 01:37:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 646.899434ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........] duration: 11.958565516s FAIL (2025-07-28 01:37:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 11.958601604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................X..XX.X..X.X...........] duration: 540.123805ms FAIL (2025-07-28 01:37:05 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 540.164823ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X............X..............................] duration: 237.244492ms FAIL (2025-07-28 01:36:41 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 237.340239ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [....XX........X..X...........X.X....X.XX....................] duration: 436.067474ms FAIL (2025-07-28 01:36:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 436.102722ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X............X.............................] duration: 263.563669ms FAIL (2025-07-28 01:36:12 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 263.663832ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........] duration: 9.47373856s FAIL (2025-07-28 01:36:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 9.473795833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................X..XX.X..X.X..........] duration: 488.49799ms FAIL (2025-07-28 01:35:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 488.535425ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................X............X............................] duration: 228.254761ms FAIL (2025-07-28 01:35:46 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 228.289039ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration: 13.209128778s FAIL (2025-07-28 01:35:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 13.209157487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................................X..XX.X..X.X.........] duration: 462.787034ms FAIL (2025-07-28 01:35:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.828294ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [X....XX........X..X...........X.X....X.XX...................] duration: 457.655511ms FAIL (2025-07-28 01:35:21 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 457.684383ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X............X...........................] duration: 195.102612ms FAIL (2025-07-28 01:35:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 195.131261ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................X..XX.X..X.X........] duration: 526.902744ms FAIL (2025-07-28 01:35:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 527.029903ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration: 13.414267359s FAIL (2025-07-28 01:34:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 13.414299225s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)