Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X.] duration: 1.859897059s FAIL (2025-07-27 00:12:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.859928615s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X] duration: 30.029744931s FAIL (2025-07-27 00:12:02 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 314542506 (2h19m48.095475886s) [upload=2145]: Done starting upload Jul 27 00:11:32.314833287 (2h19m48.095766663s) [upload=2145]: Uploading segments... Jul 27 00:11:32.314893632 (2h19m48.095827017s) [upload=2145]: Got next segment. Inline: false Jul 27 00:11:32.314934607 (2h19m48.095867986s) [upload=2145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:11:32.314940536 (2h19m48.095873952s) [upload=2145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:11:32.358989266 (2h19m48.139922631s) [seg_pos={0 0}, upload=2145]: Begin upload segment... Jul 27 00:11:32.359080795 (2h19m48.14001416s) [seg_pos={0 0}, upload=2145]: Done begin upload segment. Jul 27 00:11:32.359112107 (2h19m48.140045483s) [noise=true , address=45.91.116.14:4189 , piece_id=CSFAV42VOI5QRMMKSV7QDVO2TKFCM6Z7IYQAGOM5PNNQXCBCBRFA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359131009 (2h19m48.140064388s) [noise=true , address=104.36.82.9:4007 , piece_id=MAMJSEK3LWTZFU4SVWB423H4CHY4PKSPWXMVPXHIJXBFCVJISFTA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359557235 (2h19m48.14049062s) [noise=true , address=77.90.31.14:4662 , piece_id=P36FVJKX2LB2KOZZ2DNDLPWTKWFZH25DL4I5ECU4H46TEFHYODUA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359672407 (2h19m48.140605785s) [noise=true , address=104.36.82.4:4802 , piece_id=U2UGSTTSZIJD4KXJHP74UQDAGUTLJJR3OUAUM6QKU4NXZ6XBTLJA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359868527 (2h19m48.140801904s) [noise=true , address=104.36.82.6:4875 , piece_id=5Y7PJY2MRASMGJ6FNOLJNX7SJBWYHEKHIZUUEHFYD245JQQJQWOQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.360315728 (2h19m48.141249109s) [noise=true , address=104.36.82.3:4767 , piece_id=OXF6W345J543R3ECXVOP2B3TZ5RJCR4TD2LOD4FA6BWXQGONGL2Q, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.360940298 (2h19m48.141873678s) [noise=true , address=104.36.82.4:4801 , piece_id=CWHLGRRJWKGWVCNHEH3KV724JZQHMB4N26KVCHSHLKUL6VXMN5CA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363171520 (2h19m48.144104901s) [noise=true , address=45.91.116.12:4225 , piece_id=JJMGQYWNJNZPPHXAIOCCB5ZSCCL2C6F5Q43OIFUMFYXCEG5RDCRQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363609540 (2h19m48.144542915s) [noise=true , address=104.36.82.11:4113 , piece_id=6ZNSFERPUCAOXXICJKRIL62AB5TQIYEP7VZ5MUFP36XDLXR5OMBA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363934473 (2h19m48.144867872s) [noise=true , address=77.90.31.12:4590 , piece_id=NS54QU2R3THW2NDGL67OYIYBSEFFAEPBGMJE3C7W5QHMXE7NDGRQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.364418032 (2h19m48.145351412s) [noise=true , address=104.36.82.13:4186 , piece_id=S3C5O57CE4IMVVQVWAYGDNVBJHSW4WRMYSAPMIXE6SL3PZ3LHYIQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.364847636 (2h19m48.145781026s) [noise=true , address=77.90.31.18:4805 , piece_id=RUFOLEREXO44OUQYZK7ELTSM7NPQEPPRXU65BQEA3Z2ODAZ7QJFA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365229070 (2h19m48.146162462s) [noise=true , address=77.90.31.19:4842 , piece_id=IWKPAHR4I4YCPWGSNJORN3ZZ76SR2ROC5NNYUHFJZGTO3PV6XLZA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365580258 (2h19m48.146513638s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4A4S3PANYPRSQNKTJVWET2OFINLFHNG7QEU2I47REPNSIPUNY7A, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365888285 (2h19m48.146821663s) [noise=true , address=104.36.82.5:4839 , piece_id=3NGQM7LNMA36RB6MI6T2EK3TEOP2MV5THOKGNQCZSJMHSQC2O4ZQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.366216950 (2h19m48.147150328s) [noise=true , address=77.90.31.16:4735 , piece_id=DT3Z2CODEPADAJM643RFK67YOTS4E65KJUXNN4UVJX6ZV7WZM32A, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.435644764 (2h19m48.216578142s) [upload=2145]: Next returned nil segment Jul 27 00:11:32.435666271 (2h19m48.216599637s) [upload=2145]: Waiting for error group managing segments... Jul 27 00:11:32.440416261 (2h19m48.221349639s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4A4S3PANYPRSQNKTJVWET2OFINLFHNG7QEU2I47REPNSIPUNY7A, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440451732 (2h19m48.221385102s) [noise=true , address=45.91.116.12:4225 , piece_id=JJMGQYWNJNZPPHXAIOCCB5ZSCCL2C6F5Q43OIFUMFYXCEG5RDCRQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440950422 (2h19m48.221883793s) [noise=true , address=77.90.31.14:4662 , piece_id=P36FVJKX2LB2KOZZ2DNDLPWTKWFZH25DL4I5ECU4H46TEFHYODUA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440987609 (2h19m48.221920979s) [noise=true , address=77.90.31.16:4735 , piece_id=DT3Z2CODEPADAJM643RFK67YOTS4E65KJUXNN4UVJX6ZV7WZM32A, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.848839884 (2h19m48.629773253s) [noise=true , address=45.91.116.14:4189 , piece_id=CSFAV42VOI5QRMMKSV7QDVO2TKFCM6Z7IYQAGOM5PNNQXCBCBRFA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:33.836650446 (2h19m49.617583807s) [noise=true , address=77.90.31.19:4842 , piece_id=IWKPAHR4I4YCPWGSNJORN3ZZ76SR2ROC5NNYUHFJZGTO3PV6XLZA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:33.837905779 (2h19m49.618839149s) [noise=true , address=77.90.31.12:4590 , piece_id=NS54QU2R3THW2NDGL67OYIYBSEFFAEPBGMJE3C7W5QHMXE7NDGRQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:12:02.285759366 (2h20m18.066692752s) [noise=true , address=104.36.82.11:4113 , piece_id=6ZNSFERPUCAOXXICJKRIL62AB5TQIYEP7VZ5MUFP36XDLXR5OMBA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285860565 (2h20m18.066793953s) [noise=true , address=104.36.82.5:4839 , piece_id=3NGQM7LNMA36RB6MI6T2EK3TEOP2MV5THOKGNQCZSJMHSQC2O4ZQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285936009 (2h20m18.066869378s) [noise=true , address=104.36.82.9:4007 , piece_id=MAMJSEK3LWTZFU4SVWB423H4CHY4PKSPWXMVPXHIJXBFCVJISFTA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285991545 (2h20m18.066924913s) [noise=true , address=104.36.82.4:4802 , piece_id=U2UGSTTSZIJD4KXJHP74UQDAGUTLJJR3OUAUM6QKU4NXZ6XBTLJA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286048764 (2h20m18.066982132s) [noise=true , address=104.36.82.6:4875 , piece_id=5Y7PJY2MRASMGJ6FNOLJNX7SJBWYHEKHIZUUEHFYD245JQQJQWOQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286105406 (2h20m18.067038773s) [noise=true , address=104.36.82.3:4767 , piece_id=OXF6W345J543R3ECXVOP2B3TZ5RJCR4TD2LOD4FA6BWXQGONGL2Q, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286160478 (2h20m18.067093844s) [noise=true , address=104.36.82.4:4801 , piece_id=CWHLGRRJWKGWVCNHEH3KV724JZQHMB4N26KVCHSHLKUL6VXMN5CA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286228850 (2h20m18.067162216s) [noise=true , address=104.36.82.13:4186 , piece_id=S3C5O57CE4IMVVQVWAYGDNVBJHSW4WRMYSAPMIXE6SL3PZ3LHYIQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286292018 (2h20m18.067225381s) [noise=true , address=77.90.31.18:4805 , piece_id=RUFOLEREXO44OUQYZK7ELTSM7NPQEPPRXU65BQEA3Z2ODAZ7QJFA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 159.223.143.39:34508->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286313912 (2h20m18.067247312s) [seg_pos={0 0}, upload=2145]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 27 00:12:02.311015245 (2h20m18.091948638s) [upload=2145]: Done uploading segments... Jul 27 00:12:02.311076840 (2h20m18.092010226s) [upload=2145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.029802338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X] duration: 30.032137158s FAIL (2025-07-27 00:12:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.032167336s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X....XX..............X.X........X...XX...................] duration: 27.890976862s FAIL (2025-07-27 00:12:01 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.890994989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.025346953s FAIL (2025-07-27 00:11:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 7A, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.881067654 (2h19m14.662001045s) [noise=true , address=104.36.82.40:4959 , piece_id=O2CPROZZVXGEJ6YKX2LCBSNC2P6Y4KMPKTACKT2PZUVLA3B3VDBA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.881452462 (2h19m14.662385852s) [noise=true , address=104.36.82.18:4399 , piece_id=GNDN3YWCUOYRWNZUUMVEMKKM5H7SQ7IQ5JWKARZ7OP3KSLHDHBHA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.881819908 (2h19m14.662753296s) [noise=true , address=104.36.82.42:4805 , piece_id=PSMYKJWVGDAFPEVQMOGZOJFR4ZYT7K7MCCBPEFP6E3BTS5CWHB3A, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.882190364 (2h19m14.663123756s) [noise=true , address=104.36.82.46:4514 , piece_id=HNYGBOI56NAFJROB3EWRJOYUA4XK5GO6VNKCQLBWFFJKM6ZCHRLA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.882556447 (2h19m14.663489836s) [noise=true , address=104.36.82.56:4249 , piece_id=2ZWUAGX76VFWDJQPCSMJJRTMZXDIQJJ4XW4Y4SRGAKL4HVWI3HQA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.882973579 (2h19m14.66390696s) [noise=true , address=104.36.82.47:4535 , piece_id=PR7H75DZ6HZ5I6SILCR4EE6K57P6OYJKMZD44OYZVA2HFF6A5PFA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.883384581 (2h19m14.664317958s) [noise=true , address=104.36.82.55:4215 , piece_id=P6ZNQ2F2RPRYNPGO3ROV6VCNW3UN6H7GG3M2L2DRLMRZVKY4NMGQ, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.883873128 (2h19m14.664806515s) [noise=true , address=79.127.205.233:4639 , piece_id=FOZNY7FXI2QCJRR7OZU26AHY2JX2W5MLMWETM2QZSNX7SDTSBU7Q, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.884495674 (2h19m14.665429051s) [noise=true , address=79.127.219.38:4015 , piece_id=XL2ITS4UWY5ODJUWRRC2BXBV6S6YNECZE2FZPXN2JH2SNMEXR3GQ, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.884972886 (2h19m14.665906273s) [noise=true , address=104.36.82.54:4171 , piece_id=UHBGTYY72VGGPTBRDO7ZQWVYCF7ZZNCUVF6YHGZK7TDQRU3JLRBA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.885408826 (2h19m14.666342216s) [noise=true , address=77.90.31.10:4528 , piece_id=5FUET73FWANJX6M55UBWQFTUUW7DR33Y525QV5UIKQK2ZDIFT3AA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.885896278 (2h19m14.666829671s) [noise=true , address=79.127.201.215:4350 , piece_id=2E4HNJHTK4E3OQ7DF4SWRNUUYTMOEBT5WCIQ664AFOR52JDL6RMQ, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.886837619 (2h19m14.667770997s) [noise=true , address=64.49.13.140:4514 , piece_id=BPBPL5LQO6NG4ONEXX72GOLAGLGSIJ4PXLBOVBOK7XNEWSYX7X7A, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.887260539 (2h19m14.66819392s) [noise=true , address=104.36.82.52:4105 , piece_id=2ME3LRSUZEW2W6NZSZQNSWDF6N6F2ISULK6DGBCHXCGMV2ZV4MQQ, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.887630196 (2h19m14.668563577s) [noise=true , address=77.90.31.16:4739 , piece_id=TY2XBOMIVHR2FGKSPXBUF3INBH3YGCR2YQKR4PL5J2V72Y4SPQVQ, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.980653446 (2h19m14.761586835s) [upload=2135]: Next returned nil segment Jul 27 00:10:58.980678763 (2h19m14.761612152s) [upload=2135]: Waiting for error group managing segments... Jul 27 00:10:58.997545538 (2h19m14.778478919s) [noise=true , address=77.90.31.10:4528 , piece_id=5FUET73FWANJX6M55UBWQFTUUW7DR33Y525QV5UIKQK2ZDIFT3AA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:10:59.017538436 (2h19m14.7984718s) [noise=true , address=77.90.31.16:4739 , piece_id=TY2XBOMIVHR2FGKSPXBUF3INBH3YGCR2YQKR4PL5J2V72Y4SPQVQ, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:10:59.073488736 (2h19m14.854422106s) [noise=true , address=79.127.219.38:4015 , piece_id=XL2ITS4UWY5ODJUWRRC2BXBV6S6YNECZE2FZPXN2JH2SNMEXR3GQ, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:10:59.258126114 (2h19m15.039059503s) [noise=true , address=64.49.13.140:4514 , piece_id=BPBPL5LQO6NG4ONEXX72GOLAGLGSIJ4PXLBOVBOK7XNEWSYX7X7A, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 27 00:10:59.525879494 (2h19m15.306812857s) [noise=true , address=79.127.205.233:4639 , piece_id=FOZNY7FXI2QCJRR7OZU26AHY2JX2W5MLMWETM2QZSNX7SDTSBU7Q, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:11:01.690030080 (2h19m17.470963465s) [noise=true , address=79.127.220.101:4048 , piece_id=SWQK3DG2OGZZN3Z4JNKGFAWAOPHHK74WM5JMGGJBN6FRWGU2HOHA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:11:03.773632824 (2h19m19.554566206s) [noise=true , address=77.90.31.19:4871 , piece_id=MDPGZ5MUGDD5H2XNE6PV7X73R5TQOXBDU6RSM4CZTECRF44LB67A, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:11:20.458380627 (2h19m36.239313989s) [noise=true , address=79.127.201.215:4350 , piece_id=2E4HNJHTK4E3OQ7DF4SWRNUUYTMOEBT5WCIQ664AFOR52JDL6RMQ, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:11:58.811887908 (2h20m14.592821273s) [noise=true , address=104.36.82.18:4399 , piece_id=GNDN3YWCUOYRWNZUUMVEMKKM5H7SQ7IQ5JWKARZ7OP3KSLHDHBHA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1QPwXaFGpwMLXMiSnkE1eycqZkUSnDYMj9sFYZ2BPrPbenw9uK, address:104.36.82.18:4399): protocol: rpc: dial tcp 104.36.82.18:4399: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.811962735 (2h20m14.592896101s) [noise=true , address=104.36.82.55:4215 , piece_id=P6ZNQ2F2RPRYNPGO3ROV6VCNW3UN6H7GG3M2L2DRLMRZVKY4NMGQ, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812055287 (2h20m14.592988678s) [noise=true , address=104.36.82.54:4171 , piece_id=UHBGTYY72VGGPTBRDO7ZQWVYCF7ZZNCUVF6YHGZK7TDQRU3JLRBA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: rpc: dial tcp 104.36.82.54:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812318752 (2h20m14.593252132s) [noise=true , address=104.36.82.40:4959 , piece_id=O2CPROZZVXGEJ6YKX2LCBSNC2P6Y4KMPKTACKT2PZUVLA3B3VDBA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1NCN7sTw12KVDpD8zLjUxQki2MdMuDJyEnPSirG8ni5L2xi7aa, address:104.36.82.40:4959): protocol: rpc: dial tcp 104.36.82.40:4959: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812410763 (2h20m14.59334414s) [noise=true , address=104.36.82.42:4805 , piece_id=PSMYKJWVGDAFPEVQMOGZOJFR4ZYT7K7MCCBPEFP6E3BTS5CWHB3A, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1nh6Kih617STw2M8iHVjs4xP1HjjckJHyUxQ5z6ZFRhWwG6RXi, address:104.36.82.42:4805): protocol: rpc: dial tcp 104.36.82.42:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812478404 (2h20m14.593411779s) [noise=true , address=104.36.82.52:4105 , piece_id=2ME3LRSUZEW2W6NZSZQNSWDF6N6F2ISULK6DGBCHXCGMV2ZV4MQQ, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:104.36.82.52:4105): protocol: rpc: dial tcp 104.36.82.52:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812487111 (2h20m14.593420472s) [noise=true , address=104.36.82.47:4535 , piece_id=PR7H75DZ6HZ5I6SILCR4EE6K57P6OYJKMZD44OYZVA2HFF6A5PFA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812533012 (2h20m14.593466388s) [noise=true , address=104.36.82.46:4514 , piece_id=HNYGBOI56NAFJROB3EWRJOYUA4XK5GO6VNKCQLBWFFJKM6ZCHRLA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: rpc: dial tcp 104.36.82.46:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812553071 (2h20m14.593486437s) [noise=true , address=104.36.82.56:4249 , piece_id=2ZWUAGX76VFWDJQPCSMJJRTMZXDIQJJ4XW4Y4SRGAKL4HVWI3HQA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: write tcp 159.223.143.39:43292->104.36.82.56:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812576055 (2h20m14.593509439s) [noise=true , address=104.36.82.11:4146 , piece_id=F26GRGJBPE72Y65P6GHG4KSAHOQGTCKRJ4VGMIFHULZWJUEYSJ7A, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1Sv7eCy62QTiVT1c7mxzJSTAnrCsDoRrxqxfL6govEFRJRQJ6V, address:104.36.82.11:4146): protocol: rpc: dial tcp 104.36.82.11:4146: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812597995 (2h20m14.593531381s) [seg_pos={0 0}, upload=2135]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:11:58.835232957 (2h20m14.616166336s) [upload=2135]: Done uploading segments... Jul 27 00:11:58.835258332 (2h20m14.616191717s) [upload=2135]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.025362491s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX...XX.....X.XXX...............X..X......X.XX....X...X.....] duration: 643.75865ms FAIL (2025-07-27 00:11:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 643.853753ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X...X....XX..............X.X........X...XX..................] duration: 36.047153233s FAIL (2025-07-27 00:11:33 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.047193027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029669534s FAIL (2025-07-27 00:10:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 00:09:58.870609219 (2h18m14.651542604s) [noise=true , address=156.146.43.225:4449 , piece_id=ED56GBHDKQKYJYJUXCEWJXZ2SE42ZBNOVE2ESHN57KQJMA7RNTXA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.870621625 (2h18m14.651555001s) [noise=true , address=104.36.82.46:4497 , piece_id=G7QQZJ5BY7VEOGYGNBDJRG5FB7MEL66HKNHMXUQ2IKU7FV53FYFQ, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871073095 (2h18m14.652006476s) [noise=true , address=79.127.205.233:4645 , piece_id=XNZHWR6TWYKD7PU6M7UZYYYN2ZV6N4NFH7O7DIABHT3J547CNK4A, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871085543 (2h18m14.652018924s) [noise=true , address=77.90.31.18:4823 , piece_id=JXXOOKSXTFU52RFTWR6N4HJGP2EDABRLY54DIEKLCGJ73T2UB5AA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871342139 (2h18m14.652275521s) [noise=true , address=104.36.82.47:4529 , piece_id=633WRSG6WGI7TFOEAODZEVW2IA3ECPTXBBBLF6IOJTEATZRHQKWA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871466604 (2h18m14.652399982s) [noise=true , address=104.36.82.55:4214 , piece_id=WONYGNS4GCOIVA4H5MQEU4RUPX6CR5WF73VJSHECWLYFQWHLOB2A, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871587034 (2h18m14.652520415s) [noise=true , address=104.36.82.53:4140 , piece_id=2RN6O4BKSKG446J5EJ7MRM5XLGN33BLR3UQEWWY6BCWI2QSZ5JWQ, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871827370 (2h18m14.652760731s) [noise=true , address=104.36.82.56:4251 , piece_id=SFRKCK6KDDQHVX2FLUMGWRYCGSGL5DIG43RJYMBGAUDB6S3SYPEA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871841213 (2h18m14.652774591s) [noise=true , address=79.127.205.246:4285 , piece_id=WGIZ4OG7LDIZL2SOMQAUYTWC333UPRE5UDGV5M2PSYVTOOOWCL3A, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.872348907 (2h18m14.653282288s) [noise=true , address=77.90.31.10:4536 , piece_id=4KQKXHZUXDUOAPA2UWGMBWLAZSQE225IEWLTZLFY6DHPKVAUGIFA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.872738965 (2h18m14.653672343s) [noise=true , address=79.127.201.215:4348 , piece_id=S4YE4KIUI7Y4UH7WQVS7PTFAA6FT7BAOETDUCZ64LA6E7OTXEM2A, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.873122358 (2h18m14.65405574s) [noise=true , address=45.91.116.9:4277 , piece_id=YOOP7ICN6VJM33ROHABESV7F45QWWIVZ2C5UCXQZBF2OSPUYKUGA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.873787435 (2h18m14.6547208s) [noise=true , address=79.127.219.38:4015 , piece_id=V5S4T4YYBFIH5XOX3OFU3TPCQSBC2NHQA6ZVOX3GP7PVRBU6YKXQ, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.874096515 (2h18m14.655029901s) [noise=true , address=64.49.13.26:4953 , piece_id=SGQUJKG46EFTE63625ZPQ5ALK2SP6RSQX7YH2SO4OYL5K5IO5JEQ, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.874995372 (2h18m14.655928757s) [noise=true , address=79.127.205.234:4607 , piece_id=357BKCQLOJI25YBAGTUMBGF6EB2734SZDHZTBVQCYTSLILZ5NIGA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.876171222 (2h18m14.657104609s) [noise=true , address=104.36.82.51:4062 , piece_id=FURTWHCQJJ5BTGZUNXSR3LZIXKIDZX6AO4VOUGM6GO55VIJMBHQQ, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.877288285 (2h18m14.65822167s) [noise=true , address=79.127.205.245:4247 , piece_id=OTD7OZKXNXQT4AAGZ4KTTKQWGBMAYLV6WQKNZ32I42K5DH7QQNPA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.878143783 (2h18m14.659077173s) [noise=true , address=45.91.116.14:4194 , piece_id=VMSG5JXP4QHLTBIPEFBZCYLILAW76LF2U5ESVRV5MLMJ7RZWGQBA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.954651744 (2h18m14.735585121s) [upload=2117]: Next returned nil segment Jul 27 00:09:58.954674071 (2h18m14.735607454s) [upload=2117]: Waiting for error group managing segments... Jul 27 00:09:58.981385914 (2h18m14.762319282s) [noise=true , address=79.127.219.38:4015 , piece_id=V5S4T4YYBFIH5XOX3OFU3TPCQSBC2NHQA6ZVOX3GP7PVRBU6YKXQ, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:09:58.995594996 (2h18m14.776528375s) [noise=true , address=77.90.31.10:4536 , piece_id=4KQKXHZUXDUOAPA2UWGMBWLAZSQE225IEWLTZLFY6DHPKVAUGIFA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:09:59.444828594 (2h18m15.22576197s) [noise=true , address=79.127.205.233:4645 , piece_id=XNZHWR6TWYKD7PU6M7UZYYYN2ZV6N4NFH7O7DIABHT3J547CNK4A, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:09:59.473968428 (2h18m15.254901804s) [noise=true , address=79.127.205.246:4285 , piece_id=WGIZ4OG7LDIZL2SOMQAUYTWC333UPRE5UDGV5M2PSYVTOOOWCL3A, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:09:59.485903479 (2h18m15.266836857s) [noise=true , address=79.127.205.245:4247 , piece_id=OTD7OZKXNXQT4AAGZ4KTTKQWGBMAYLV6WQKNZ32I42K5DH7QQNPA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:09:59.767063697 (2h18m15.547997077s) [noise=true , address=79.127.205.234:4607 , piece_id=357BKCQLOJI25YBAGTUMBGF6EB2734SZDHZTBVQCYTSLILZ5NIGA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:10:03.561863986 (2h18m19.342797355s) [noise=true , address=156.146.43.225:4449 , piece_id=ED56GBHDKQKYJYJUXCEWJXZ2SE42ZBNOVE2ESHN57KQJMA7RNTXA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:10:04.426823907 (2h18m20.207757282s) [noise=true , address=45.91.116.14:4194 , piece_id=VMSG5JXP4QHLTBIPEFBZCYLILAW76LF2U5ESVRV5MLMJ7RZWGQBA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 27 00:10:05.779230864 (2h18m21.560164233s) [noise=true , address=64.49.13.26:4953 , piece_id=SGQUJKG46EFTE63625ZPQ5ALK2SP6RSQX7YH2SO4OYL5K5IO5JEQ, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:10:08.236128125 (2h18m24.017061513s) [noise=true , address=45.91.116.9:4277 , piece_id=YOOP7ICN6VJM33ROHABESV7F45QWWIVZ2C5UCXQZBF2OSPUYKUGA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:10:35.516470487 (2h18m51.297403865s) [noise=true , address=79.127.201.215:4348 , piece_id=S4YE4KIUI7Y4UH7WQVS7PTFAA6FT7BAOETDUCZ64LA6E7OTXEM2A, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:10:58.708431972 (2h19m14.489365337s) [noise=true , address=104.36.82.53:4140 , piece_id=2RN6O4BKSKG446J5EJ7MRM5XLGN33BLR3UQEWWY6BCWI2QSZ5JWQ, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12Wv1U4bCkzg9Ywx4EKRkWG8tMffjwiVBEZ6tHr8scNHfdcWyrL, address:104.36.82.53:4140): protocol: rpc: dial tcp 104.36.82.53:4140: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708543339 (2h19m14.489476717s) [noise=true , address=77.90.31.18:4823 , piece_id=JXXOOKSXTFU52RFTWR6N4HJGP2EDABRLY54DIEKLCGJ73T2UB5AA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708621455 (2h19m14.489554842s) [noise=true , address=104.36.82.51:4062 , piece_id=FURTWHCQJJ5BTGZUNXSR3LZIXKIDZX6AO4VOUGM6GO55VIJMBHQQ, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: write tcp 159.223.143.39:37192->104.36.82.51:4062: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708683851 (2h19m14.48961722s) [noise=true , address=104.36.82.55:4214 , piece_id=WONYGNS4GCOIVA4H5MQEU4RUPX6CR5WF73VJSHECWLYFQWHLOB2A, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi, address:104.36.82.55:4214): protocol: write tcp 159.223.143.39:44640->104.36.82.55:4214: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708733161 (2h19m14.489666524s) [noise=true , address=104.36.82.46:4497 , piece_id=G7QQZJ5BY7VEOGYGNBDJRG5FB7MEL66HKNHMXUQ2IKU7FV53FYFQ, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12Zg4PSYBuLxFvF9oYiuvn65JNbjT5RdagcVAQLp2izvjPe1NEc, address:104.36.82.46:4497): protocol: write tcp 159.223.143.39:51100->104.36.82.46:4497: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708755188 (2h19m14.489688552s) [noise=true , address=104.36.82.47:4529 , piece_id=633WRSG6WGI7TFOEAODZEVW2IA3ECPTXBBBLF6IOJTEATZRHQKWA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:1m7FrQEps7d8enRYk4v17UxcDs8o8L2CFcVEyB3zR4EniXrYeH, address:104.36.82.47:4529): protocol: write tcp 159.223.143.39:45894->104.36.82.47:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708784893 (2h19m14.489718261s) [noise=true , address=104.36.82.56:4251 , piece_id=SFRKCK6KDDQHVX2FLUMGWRYCGSGL5DIG43RJYMBGAUDB6S3SYPEA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708841198 (2h19m14.489774585s) [seg_pos={0 0}, upload=2117]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:10:58.735372401 (2h19m14.516305779s) [upload=2117]: Done uploading segments... Jul 27 00:10:58.735424354 (2h19m14.516357734s) [upload=2117]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.029690923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X...X....XX..............X.X........X...XX.................] duration: 49.239530897s FAIL (2025-07-27 00:10:57 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.239609603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXX...XX.....X.XXX...............X..X......X.XX....X...X....] duration: 703.311818ms FAIL (2025-07-27 00:10:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 703.343065ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....] duration: 2.63219441s FAIL (2025-07-27 00:10:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.632250455s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..] duration: 17.699187622s FAIL (2025-07-27 00:10:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== _id=OTZ6FQJRXFS3GWQIC2OZKBS4GR56LPUISPEDHWWQP7EONXTK4RZQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740998689 (2h16m35.521932116s) [noise=true , address=77.90.31.15:4699 , piece_id=LFCQEJG76VOFKXE2ZCGRXL7W2VNJL4TU4JYZU57F4DXJ55M7AYTQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.741048351 (2h16m35.52198174s) [seg_pos={0 0}, upload=2083]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:08:19.764313990 (2h16m35.545247374s) [upload=2083]: Done uploading segments... Jul 27 00:08:19.764346822 (2h16m35.545280207s) [upload=2083]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:08:45.635298021 (2h17m1.4162315s) [upload=2097]: Starting upload Jul 27 00:08:45.635390125 (2h17m1.416323529s) [upload=2097]: Done starting upload Jul 27 00:08:45.635618718 (2h17m1.416552096s) [upload=2097]: Uploading segments... Jul 27 00:08:45.635780082 (2h17m1.416713462s) [upload=2097]: Got next segment. Inline: false Jul 27 00:08:45.635855437 (2h17m1.416788814s) [upload=2097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:08:45.635863070 (2h17m1.416796461s) [upload=2097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:08:45.666562989 (2h17m1.447496374s) [seg_pos={0 0}, upload=2097]: Begin upload segment... Jul 27 00:08:45.666633979 (2h17m1.447567347s) [seg_pos={0 0}, upload=2097]: Done begin upload segment. Jul 27 00:08:45.666674450 (2h17m1.447607829s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666685892 (2h17m1.447619261s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666754668 (2h17m1.447688044s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667156517 (2h17m1.448089903s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667216106 (2h17m1.448149484s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667535834 (2h17m1.448469218s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667719777 (2h17m1.448653163s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667900825 (2h17m1.448834217s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668432972 (2h17m1.449366357s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668922889 (2h17m1.449856273s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671253980 (2h17m1.452187358s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671603059 (2h17m1.452536438s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671890649 (2h17m1.452824036s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673442036 (2h17m1.454375416s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673811853 (2h17m1.454745223s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.686682455 (2h17m1.467615818s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.771440822 (2h17m1.552374209s) [upload=2097]: Next returned nil segment Jul 27 00:08:45.771463691 (2h17m1.55239708s) [upload=2097]: Waiting for error group managing segments... Jul 27 00:08:45.775342955 (2h17m1.556276318s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.776475944 (2h17m1.557409313s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777015261 (2h17m1.557948628s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777591233 (2h17m1.55852462s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778018122 (2h17m1.558951495s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778418119 (2h17m1.559351488s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778929404 (2h17m1.559862779s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779537556 (2h17m1.560470931s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779812214 (2h17m1.560745582s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:46.684702430 (2h17m2.465635798s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.073910770 (2h17m2.854844136s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.166542112 (2h17m2.947475475s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:09:15.606542690 (2h17m31.387476056s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606697150 (2h17m31.387630526s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606735183 (2h17m31.387668552s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:37514->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606756595 (2h17m31.387689958s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606778960 (2h17m31.387712338s) [seg_pos={0 0}, upload=2097]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:15.635367371 (2h17m31.416300738s) [upload=2097]: Done uploading segments... Jul 27 00:09:15.635393090 (2h17m31.416326454s) [upload=2097]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 17.699222667s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XXX...XX.....X.XXX...............X..X......X.XX....X...X...] duration: 690.33045ms FAIL (2025-07-27 00:10:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 690.44241ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX...] duration: 1.927472761s FAIL (2025-07-27 00:10:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.927584006s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X...X....XX..............X.X........X...XX................] duration: 28.575118033s FAIL (2025-07-27 00:10:07 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.57514105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027240879s FAIL (2025-07-27 00:09:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 49.13.138:4483 , piece_id=KKN7VFWOFSS6UKGMVP4O5SP34MM2KZDOCRQH7QFGD4YFQIE6YKDA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.693844258 (2h17m14.474777655s) [noise=true , address=77.90.31.15:4730 , piece_id=WVSI5HFXAUSKHLRAN4CASOGODTD4X73RDCOS4GRLRTNVUREUBY7A, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.694569395 (2h17m14.475502783s) [noise=true , address=77.90.31.15:4714 , piece_id=3HZWM765U3XAABGE3NTW5JYHDVQNRXOIKUJU2R67FFYFZIWPMYKA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.695217237 (2h17m14.47615063s) [noise=true , address=45.91.116.14:4206 , piece_id=H5BFYVDSTQKN2ZPRNOPFPS4EEUO6UY5H425OE4346QCHBBSL53CQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.695781767 (2h17m14.476715145s) [noise=true , address=104.36.82.54:4169 , piece_id=PKI7VE3YP6M6NMCW4IA5UU7RMTEL4WLATOGAHAJAOYOPTWTVJJJQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.696331308 (2h17m14.477264688s) [noise=true , address=77.90.31.19:4856 , piece_id=VQQMI4YAFAPADAXDQWUUWVGMRSYEYBV2LCKXFBBOAEKBKSZ6TNWQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.696780964 (2h17m14.477714348s) [noise=true , address=104.36.82.52:4107 , piece_id=C7RYDOFWHJSWRXUKO56UXWXTLZ5KNXKQFTAHIOQ5OIP34ESWS7PQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.697383991 (2h17m14.478317386s) [noise=true , address=79.127.205.245:4227 , piece_id=R33XV7QC723PQWKM5PR6QTVYBDBGUHVVMXQKM5ROSZ77W5J4JSAA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.697999609 (2h17m14.478932999s) [noise=true , address=104.36.82.56:4234 , piece_id=JX7MT6BQCEATXGPPHXGGGZTWPMS4MXURILRAH4TUBA6QJIPGCHNQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.698573705 (2h17m14.47950709s) [noise=true , address=79.127.219.48:4074 , piece_id=RCKEXJFHLJTFCEK5CTIFWIPTT4OG7CCAHWJ5FZWVP2BMV6MF2ADA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.699167283 (2h17m14.480100673s) [noise=true , address=64.49.13.138:4482 , piece_id=A64AWGLBYK4RANPOSQDPEFFL5TNJ6HWRPIEPDYY4IT3DZM5J53ZQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.699714793 (2h17m14.480648181s) [noise=true , address=104.36.82.55:4202 , piece_id=DPQYMMBYQZ4DM5IQLMZCIKF4HCGPAWRIVMDO3VPQ6SR4ZX37JIUQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.705705924 (2h17m14.486639308s) [noise=true , address=79.127.205.246:4281 , piece_id=WJ7SXHUVGEE36O7IKYTEUID3MBQ4BT6WEGSR5GOZ24WCOBF3FSPA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.706439516 (2h17m14.487372902s) [noise=true , address=104.36.82.46:4513 , piece_id=OIAQWPQOYAYG3Q2BFKVCRYZPY5ACJ5M3RXMU6VW4LL4QUGINGOXA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.706935202 (2h17m14.487868603s) [noise=true , address=77.90.31.16:4746 , piece_id=AA75PBHV2E73ARBPZNUTZV7B5GGBXZUOVRL3VCEZ2EZ7HMUM234A, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.707554227 (2h17m14.488487618s) [noise=true , address=79.127.205.235:4685 , piece_id=RVZNRHTSFABHZ54X33PQVLFLMFMRM7VLA7SCXDVI5BZNRBEJUKIA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.714121191 (2h17m14.495054578s) [noise=true , address=79.127.219.38:4001 , piece_id=O2JOUHJFKUGGFOFW6JRTQYIER5Z4DAPEOUGJCS3BTYDLERQFBEIA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.714751905 (2h17m14.49568529s) [noise=true , address=104.36.82.47:4547 , piece_id=CC42C6HD572T657IQSBJDINSSXDUOWMMF4KEUTH64IEPGM5WFAIA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.811431908 (2h17m14.592365278s) [upload=2101]: Next returned nil segment Jul 27 00:08:58.811453600 (2h17m14.592386965s) [upload=2101]: Waiting for error group managing segments... Jul 27 00:08:58.869964784 (2h17m14.650898166s) [noise=true , address=77.90.31.16:4746 , piece_id=AA75PBHV2E73ARBPZNUTZV7B5GGBXZUOVRL3VCEZ2EZ7HMUM234A, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:08:58.885637487 (2h17m14.666570865s) [noise=true , address=79.127.219.38:4001 , piece_id=O2JOUHJFKUGGFOFW6JRTQYIER5Z4DAPEOUGJCS3BTYDLERQFBEIA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:08:58.896591250 (2h17m14.677524628s) [noise=true , address=79.127.219.48:4074 , piece_id=RCKEXJFHLJTFCEK5CTIFWIPTT4OG7CCAHWJ5FZWVP2BMV6MF2ADA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:08:59.285579378 (2h17m15.066512749s) [noise=true , address=79.127.205.245:4227 , piece_id=R33XV7QC723PQWKM5PR6QTVYBDBGUHVVMXQKM5ROSZ77W5J4JSAA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:08:59.303912970 (2h17m15.084846332s) [noise=true , address=79.127.205.235:4685 , piece_id=RVZNRHTSFABHZ54X33PQVLFLMFMRM7VLA7SCXDVI5BZNRBEJUKIA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:08:59.331426127 (2h17m15.112359494s) [noise=true , address=79.127.205.246:4281 , piece_id=WJ7SXHUVGEE36O7IKYTEUID3MBQ4BT6WEGSR5GOZ24WCOBF3FSPA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:09:06.348746969 (2h17m22.129680336s) [noise=true , address=77.90.31.19:4856 , piece_id=VQQMI4YAFAPADAXDQWUUWVGMRSYEYBV2LCKXFBBOAEKBKSZ6TNWQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:09:07.955926224 (2h17m23.736859602s) [noise=true , address=64.49.13.138:4483 , piece_id=KKN7VFWOFSS6UKGMVP4O5SP34MM2KZDOCRQH7QFGD4YFQIE6YKDA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:09:09.229263909 (2h17m25.010197304s) [noise=true , address=64.49.13.138:4482 , piece_id=A64AWGLBYK4RANPOSQDPEFFL5TNJ6HWRPIEPDYY4IT3DZM5J53ZQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:09:10.481553148 (2h17m26.262486523s) [noise=true , address=45.91.116.14:4206 , piece_id=H5BFYVDSTQKN2ZPRNOPFPS4EEUO6UY5H425OE4346QCHBBSL53CQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:09:58.589645966 (2h18m14.370579343s) [noise=true , address=104.36.82.55:4202 , piece_id=DPQYMMBYQZ4DM5IQLMZCIKF4HCGPAWRIVMDO3VPQ6SR4ZX37JIUQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.589750540 (2h18m14.370683918s) [noise=true , address=77.90.31.15:4714 , piece_id=3HZWM765U3XAABGE3NTW5JYHDVQNRXOIKUJU2R67FFYFZIWPMYKA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: rpc: dial tcp 77.90.31.15:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.589821519 (2h18m14.370754894s) [noise=true , address=104.36.82.56:4234 , piece_id=JX7MT6BQCEATXGPPHXGGGZTWPMS4MXURILRAH4TUBA6QJIPGCHNQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: write tcp 159.223.143.39:44494->104.36.82.56:4234: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.589879415 (2h18m14.370812779s) [noise=true , address=104.36.82.47:4547 , piece_id=CC42C6HD572T657IQSBJDINSSXDUOWMMF4KEUTH64IEPGM5WFAIA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:12brQg9WQMKJgp1buz1wMS8W11XEwAx7RHG5h51EmLvwY5D5Hky, address:104.36.82.47:4547): protocol: write tcp 159.223.143.39:45666->104.36.82.47:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.589955032 (2h18m14.370888408s) [noise=true , address=104.36.82.52:4107 , piece_id=C7RYDOFWHJSWRXUKO56UXWXTLZ5KNXKQFTAHIOQ5OIP34ESWS7PQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: rpc: dial tcp 104.36.82.52:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.589997288 (2h18m14.370930671s) [noise=true , address=104.36.82.46:4513 , piece_id=OIAQWPQOYAYG3Q2BFKVCRYZPY5ACJ5M3RXMU6VW4LL4QUGINGOXA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 159.223.143.39:40684->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.590036227 (2h18m14.370969605s) [noise=true , address=104.36.82.54:4169 , piece_id=PKI7VE3YP6M6NMCW4IA5UU7RMTEL4WLATOGAHAJAOYOPTWTVJJJQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.590097110 (2h18m14.371030486s) [noise=true , address=77.90.31.15:4730 , piece_id=WVSI5HFXAUSKHLRAN4CASOGODTD4X73RDCOS4GRLRTNVUREUBY7A, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, address:77.90.31.15:4730): protocol: write tcp 159.223.143.39:58880->77.90.31.15:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.590128313 (2h18m14.371061676s) [seg_pos={0 0}, upload=2101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:09:58.615082988 (2h18m14.396016365s) [upload=2101]: Done uploading segments... Jul 27 00:09:58.615115575 (2h18m14.396048958s) [upload=2101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.02727003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX..] duration: 2.533230994s FAIL (2025-07-27 00:09:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.533264849s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...X...X....XX..............X.X........X...XX...............] duration: 43.473840759s FAIL (2025-07-27 00:09:39 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.473863951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX.] duration: 562.794377ms FAIL (2025-07-27 00:09:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== _id=OTZ6FQJRXFS3GWQIC2OZKBS4GR56LPUISPEDHWWQP7EONXTK4RZQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740998689 (2h16m35.521932116s) [noise=true , address=77.90.31.15:4699 , piece_id=LFCQEJG76VOFKXE2ZCGRXL7W2VNJL4TU4JYZU57F4DXJ55M7AYTQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.741048351 (2h16m35.52198174s) [seg_pos={0 0}, upload=2083]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:08:19.764313990 (2h16m35.545247374s) [upload=2083]: Done uploading segments... Jul 27 00:08:19.764346822 (2h16m35.545280207s) [upload=2083]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:08:45.635298021 (2h17m1.4162315s) [upload=2097]: Starting upload Jul 27 00:08:45.635390125 (2h17m1.416323529s) [upload=2097]: Done starting upload Jul 27 00:08:45.635618718 (2h17m1.416552096s) [upload=2097]: Uploading segments... Jul 27 00:08:45.635780082 (2h17m1.416713462s) [upload=2097]: Got next segment. Inline: false Jul 27 00:08:45.635855437 (2h17m1.416788814s) [upload=2097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:08:45.635863070 (2h17m1.416796461s) [upload=2097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:08:45.666562989 (2h17m1.447496374s) [seg_pos={0 0}, upload=2097]: Begin upload segment... Jul 27 00:08:45.666633979 (2h17m1.447567347s) [seg_pos={0 0}, upload=2097]: Done begin upload segment. Jul 27 00:08:45.666674450 (2h17m1.447607829s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666685892 (2h17m1.447619261s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666754668 (2h17m1.447688044s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667156517 (2h17m1.448089903s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667216106 (2h17m1.448149484s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667535834 (2h17m1.448469218s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667719777 (2h17m1.448653163s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667900825 (2h17m1.448834217s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668432972 (2h17m1.449366357s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668922889 (2h17m1.449856273s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671253980 (2h17m1.452187358s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671603059 (2h17m1.452536438s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671890649 (2h17m1.452824036s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673442036 (2h17m1.454375416s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673811853 (2h17m1.454745223s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.686682455 (2h17m1.467615818s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.771440822 (2h17m1.552374209s) [upload=2097]: Next returned nil segment Jul 27 00:08:45.771463691 (2h17m1.55239708s) [upload=2097]: Waiting for error group managing segments... Jul 27 00:08:45.775342955 (2h17m1.556276318s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.776475944 (2h17m1.557409313s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777015261 (2h17m1.557948628s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777591233 (2h17m1.55852462s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778018122 (2h17m1.558951495s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778418119 (2h17m1.559351488s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778929404 (2h17m1.559862779s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779537556 (2h17m1.560470931s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779812214 (2h17m1.560745582s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:46.684702430 (2h17m2.465635798s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.073910770 (2h17m2.854844136s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.166542112 (2h17m2.947475475s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:09:15.606542690 (2h17m31.387476056s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606697150 (2h17m31.387630526s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606735183 (2h17m31.387668552s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:37514->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606756595 (2h17m31.387689958s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606778960 (2h17m31.387712338s) [seg_pos={0 0}, upload=2097]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:15.635367371 (2h17m31.416300738s) [upload=2097]: Done uploading segments... Jul 27 00:09:15.635393090 (2h17m31.416326454s) [upload=2097]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 562.893069ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XXX...XX.....X.XXX...............X..X......X.XX....X...X..] duration: 483.022918ms FAIL (2025-07-27 00:09:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.044551ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX.] duration: 1.867645699s FAIL (2025-07-27 00:09:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.86769198s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX] duration: 30.031054088s FAIL (2025-07-27 00:09:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== _id=OTZ6FQJRXFS3GWQIC2OZKBS4GR56LPUISPEDHWWQP7EONXTK4RZQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740998689 (2h16m35.521932116s) [noise=true , address=77.90.31.15:4699 , piece_id=LFCQEJG76VOFKXE2ZCGRXL7W2VNJL4TU4JYZU57F4DXJ55M7AYTQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.741048351 (2h16m35.52198174s) [seg_pos={0 0}, upload=2083]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:08:19.764313990 (2h16m35.545247374s) [upload=2083]: Done uploading segments... Jul 27 00:08:19.764346822 (2h16m35.545280207s) [upload=2083]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:08:45.635298021 (2h17m1.4162315s) [upload=2097]: Starting upload Jul 27 00:08:45.635390125 (2h17m1.416323529s) [upload=2097]: Done starting upload Jul 27 00:08:45.635618718 (2h17m1.416552096s) [upload=2097]: Uploading segments... Jul 27 00:08:45.635780082 (2h17m1.416713462s) [upload=2097]: Got next segment. Inline: false Jul 27 00:08:45.635855437 (2h17m1.416788814s) [upload=2097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:08:45.635863070 (2h17m1.416796461s) [upload=2097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:08:45.666562989 (2h17m1.447496374s) [seg_pos={0 0}, upload=2097]: Begin upload segment... Jul 27 00:08:45.666633979 (2h17m1.447567347s) [seg_pos={0 0}, upload=2097]: Done begin upload segment. Jul 27 00:08:45.666674450 (2h17m1.447607829s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666685892 (2h17m1.447619261s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666754668 (2h17m1.447688044s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667156517 (2h17m1.448089903s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667216106 (2h17m1.448149484s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667535834 (2h17m1.448469218s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667719777 (2h17m1.448653163s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667900825 (2h17m1.448834217s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668432972 (2h17m1.449366357s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668922889 (2h17m1.449856273s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671253980 (2h17m1.452187358s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671603059 (2h17m1.452536438s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671890649 (2h17m1.452824036s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673442036 (2h17m1.454375416s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673811853 (2h17m1.454745223s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.686682455 (2h17m1.467615818s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.771440822 (2h17m1.552374209s) [upload=2097]: Next returned nil segment Jul 27 00:08:45.771463691 (2h17m1.55239708s) [upload=2097]: Waiting for error group managing segments... Jul 27 00:08:45.775342955 (2h17m1.556276318s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.776475944 (2h17m1.557409313s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777015261 (2h17m1.557948628s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777591233 (2h17m1.55852462s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778018122 (2h17m1.558951495s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778418119 (2h17m1.559351488s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778929404 (2h17m1.559862779s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779537556 (2h17m1.560470931s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779812214 (2h17m1.560745582s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:46.684702430 (2h17m2.465635798s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.073910770 (2h17m2.854844136s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.166542112 (2h17m2.947475475s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:09:15.606542690 (2h17m31.387476056s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606697150 (2h17m31.387630526s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606735183 (2h17m31.387668552s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:37514->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606756595 (2h17m31.387689958s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606778960 (2h17m31.387712338s) [seg_pos={0 0}, upload=2097]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:15.635367371 (2h17m31.416300738s) [upload=2097]: Done uploading segments... Jul 27 00:09:15.635393090 (2h17m31.416326454s) [upload=2097]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031091737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.035207603s FAIL (2025-07-27 00:08:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pb.BatchRequestItem_SegmentBegin Jul 27 00:07:58.571471335 (2h16m14.352404726s) [seg_pos={0 0}, upload=2086]: Begin upload segment... Jul 27 00:07:58.571528992 (2h16m14.352462372s) [seg_pos={0 0}, upload=2086]: Done begin upload segment. Jul 27 00:07:58.571560811 (2h16m14.352494193s) [noise=true , address=45.91.116.14:4206 , piece_id=PJKH4TLQXKTOCCH3GAZVCFNM54SARIP3DTL4W4R6OGWIDL6C6EBQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.571880479 (2h16m14.352813863s) [noise=true , address=79.127.205.245:4246 , piece_id=HFQKIFWQGBQIBBG5TFMSGY6SEEGVAU4BBDNVSBNINHBHLWMO4TPA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.572151788 (2h16m14.353085178s) [noise=true , address=79.127.205.234:4605 , piece_id=SZENAG67GU2AXNZ3CIOOQQD3UC7RV2SKMVJ24BHZJNUQVN6T5ZMQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.572719939 (2h16m14.353653324s) [noise=true , address=79.127.220.98:4422 , piece_id=UIKENZ43BPVLCJJ6WRQGVPZORCKGW36RKMIN23FPGSGNFKP55UJA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.572803014 (2h16m14.353736403s) [noise=true , address=79.127.205.246:4283 , piece_id=KE5SLI2DBHGYIP72NCHS7674KCFLRP5K775AJHVVEHLGARLRVIXA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.573291054 (2h16m14.354224464s) [noise=true , address=79.127.220.97:4361 , piece_id=RMYNTMR6W34NEYXLUWDNQLS4LXSHZOGMPPJTOOU3QMHEGXA2BFNA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.573300466 (2h16m14.354233848s) [noise=true , address=77.90.31.16:4743 , piece_id=OLKUPFXZCEV7FKHOVUSMNVM2RKOP2PYMZ2PUDXHLLSXKGCH2ZW2Q, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.573752384 (2h16m14.354685768s) [noise=true , address=104.36.82.53:4144 , piece_id=ZRLN4TXEOQVSBICQZFS3DPOSYOISCKSW6GCQFM42FULQM3XZG63Q, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.573885407 (2h16m14.354818793s) [noise=true , address=79.127.201.215:4351 , piece_id=5JVRJOR6A2TH7WMEE647K3APUK4XJODB23MU2HSI6KHVTK6XCZYA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.574210503 (2h16m14.355143895s) [noise=true , address=77.90.31.19:4871 , piece_id=WRAKP2KI5M5L4URDPI3KUHPCTIIHKRFNZ6VUN32H7ELEWUEMTRVQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.574428443 (2h16m14.355361825s) [noise=true , address=104.36.82.56:4257 , piece_id=OJFFPWZ2CGHSD76Z73XUIYVUITPGKYP6P7EFONOYXOGUDX45ZUIA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.574640678 (2h16m14.355574057s) [noise=true , address=45.91.116.9:4277 , piece_id=THCPO5GM4BVJPQILOVPYHOVUHESWFB4AU34Q435NRCYKNRQP54KQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.575136064 (2h16m14.356069446s) [noise=true , address=77.90.31.13:4653 , piece_id=MEFACJFVOZR27ENWMH4G7WDV2FRY36KDY6T2CCPX2XPWQJLC2U5A, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.575235618 (2h16m14.356169003s) [noise=true , address=104.36.82.46:4501 , piece_id=CHWEXONG3ZVPDQWHLCJZXIHPY6TKEWDUAKGPYR2EOXRYV4GF3DGQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.575573434 (2h16m14.356506913s) [noise=true , address=77.90.31.18:4823 , piece_id=Y5FMCBEYBTDR6QHWIIZS7YP26B2L45IXKONF6FOL2CY2CPILD5WQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.575753558 (2h16m14.356686938s) [noise=true , address=104.36.82.47:4527 , piece_id=7KD6WHTAZVDHKZLDYNGHZF7KQABMOJKY3EY3LKXGSUIJUSSFOVWQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.576239310 (2h16m14.35717269s) [noise=true , address=104.36.82.52:4114 , piece_id=VMMLOIGHHIKR3YTCHZ7Z7TGZ53J52FYY3LVJT3X3HJL4PVB7JAMQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.576771158 (2h16m14.357704539s) [noise=true , address=77.90.31.21:4909 , piece_id=DZJLMYK72PGPOWIXATBXEYBTBAGPWQV3KPTKXWVKUWKDJNEDPY6A, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.656387892 (2h16m14.437321259s) [upload=2086]: Next returned nil segment Jul 27 00:07:58.656405692 (2h16m14.437339058s) [upload=2086]: Waiting for error group managing segments... Jul 27 00:07:58.661053248 (2h16m14.441986624s) [noise=true , address=77.90.31.16:4743 , piece_id=OLKUPFXZCEV7FKHOVUSMNVM2RKOP2PYMZ2PUDXHLLSXKGCH2ZW2Q, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:07:58.689500879 (2h16m14.470434254s) [noise=true , address=79.127.220.97:4361 , piece_id=RMYNTMR6W34NEYXLUWDNQLS4LXSHZOGMPPJTOOU3QMHEGXA2BFNA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:07:58.824486404 (2h16m14.605419784s) [noise=true , address=77.90.31.18:4823 , piece_id=Y5FMCBEYBTDR6QHWIIZS7YP26B2L45IXKONF6FOL2CY2CPILD5WQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:07:58.826808475 (2h16m14.607741843s) [noise=true , address=77.90.31.21:4909 , piece_id=DZJLMYK72PGPOWIXATBXEYBTBAGPWQV3KPTKXWVKUWKDJNEDPY6A, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:07:58.911693819 (2h16m14.692627205s) [noise=true , address=45.91.116.9:4277 , piece_id=THCPO5GM4BVJPQILOVPYHOVUHESWFB4AU34Q435NRCYKNRQP54KQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:07:59.160351458 (2h16m14.941284823s) [noise=true , address=79.127.205.245:4246 , piece_id=HFQKIFWQGBQIBBG5TFMSGY6SEEGVAU4BBDNVSBNINHBHLWMO4TPA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:07:59.163461136 (2h16m14.944394513s) [noise=true , address=79.127.205.234:4605 , piece_id=SZENAG67GU2AXNZ3CIOOQQD3UC7RV2SKMVJ24BHZJNUQVN6T5ZMQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:07:59.179814556 (2h16m14.960747926s) [noise=true , address=79.127.205.246:4283 , piece_id=KE5SLI2DBHGYIP72NCHS7674KCFLRP5K775AJHVVEHLGARLRVIXA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:00.141739984 (2h16m15.922673364s) [noise=true , address=79.127.220.98:4422 , piece_id=UIKENZ43BPVLCJJ6WRQGVPZORCKGW36RKMIN23FPGSGNFKP55UJA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:02.644197060 (2h16m18.425130438s) [noise=true , address=77.90.31.19:4871 , piece_id=WRAKP2KI5M5L4URDPI3KUHPCTIIHKRFNZ6VUN32H7ELEWUEMTRVQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:06.978277581 (2h16m22.75921095s) [noise=true , address=45.91.116.14:4206 , piece_id=PJKH4TLQXKTOCCH3GAZVCFNM54SARIP3DTL4W4R6OGWIDL6C6EBQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:31.041151596 (2h16m46.822084965s) [noise=true , address=79.127.201.215:4351 , piece_id=5JVRJOR6A2TH7WMEE647K3APUK4XJODB23MU2HSI6KHVTK6XCZYA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:54.070821324 (2h17m9.851754702s) [noise=true , address=77.90.31.13:4653 , piece_id=MEFACJFVOZR27ENWMH4G7WDV2FRY36KDY6T2CCPX2XPWQJLC2U5A, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:58.485000935 (2h17m14.265934317s) [noise=true , address=104.36.82.47:4527 , piece_id=7KD6WHTAZVDHKZLDYNGHZF7KQABMOJKY3EY3LKXGSUIJUSSFOVWQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: write tcp 159.223.143.39:45842->104.36.82.47:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.485126710 (2h17m14.266060089s) [noise=true , address=104.36.82.46:4501 , piece_id=CHWEXONG3ZVPDQWHLCJZXIHPY6TKEWDUAKGPYR2EOXRYV4GF3DGQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: write tcp 159.223.143.39:47962->104.36.82.46:4501: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.486013090 (2h17m14.266946472s) [noise=true , address=104.36.82.56:4257 , piece_id=OJFFPWZ2CGHSD76Z73XUIYVUITPGKYP6P7EFONOYXOGUDX45ZUIA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:12iw1KEPBAYhvSachvkigv7bPXfGN5XEugvxGC5YTBV2Muymir3, address:104.36.82.56:4257): protocol: write tcp 159.223.143.39:51022->104.36.82.56:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.486104889 (2h17m14.267038257s) [noise=true , address=104.36.82.52:4114 , piece_id=VMMLOIGHHIKR3YTCHZ7Z7TGZ53J52FYY3LVJT3X3HJL4PVB7JAMQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:12PXReFMGqAhCAp5BSE3vsJdfRAUathsJLdB8Guz7DzW6UHmATM, address:104.36.82.52:4114): protocol: write tcp 159.223.143.39:34840->104.36.82.52:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.486174191 (2h17m14.267107557s) [noise=true , address=104.36.82.53:4144 , piece_id=ZRLN4TXEOQVSBICQZFS3DPOSYOISCKSW6GCQFM42FULQM3XZG63Q, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:1FfzcPNLmNdrvoZBKcmCLz8d9rYSAnjFTo4NBnxBzN6dVcTNwt, address:104.36.82.53:4144): protocol: write tcp 159.223.143.39:37018->104.36.82.53:4144: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.486197912 (2h17m14.267131297s) [seg_pos={0 0}, upload=2086]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:08:58.515202173 (2h17m14.296135543s) [upload=2086]: Done uploading segments... Jul 27 00:08:58.515241979 (2h17m14.296175358s) [upload=2086]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.035233581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX] duration: 30.027115385s FAIL (2025-07-27 00:08:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.027173922s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....X...X....XX..............X.X........X...XX..............] duration: 29.871730396s FAIL (2025-07-27 00:08:34 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.871763063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....X] duration: 30.027033025s FAIL (2025-07-27 00:08:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 528133661s) [noise=true , address=104.36.82.5:4837 , piece_id=K3P6BF7XD4U6ASNOX6O625GZQUCAYF6GKTNNAPJLICBHRYGZJDRA, seg_pos={0 0}, upload=2063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: rpc: dial tcp 104.36.82.5:4837: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Jul 27 00:06:38.747231402 (2h14m54.528164781s) [seg_pos={0 0}, upload=2063]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:06:38.747323302 (2h14m54.528256679s) [upload=2063]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:06:38.747340233 (2h14m54.528273609s) [upload=2063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:06:38.884500181 (2h14m54.665433546s) [upload=2063]: Done uploading segments... Jul 27 00:06:38.884518081 (2h14m54.66545145s) [upload=2063]: Upload finished. err: <nil> Jul 27 00:07:49.767137327 (2h16m5.548070693s) [upload=2083]: Starting upload Jul 27 00:07:49.767257443 (2h16m5.548190826s) [upload=2083]: Done starting upload Jul 27 00:07:49.767387204 (2h16m5.548320576s) [upload=2083]: Uploading segments... Jul 27 00:07:49.767698085 (2h16m5.548631462s) [upload=2083]: Got next segment. Inline: false Jul 27 00:07:49.767782455 (2h16m5.548715837s) [upload=2083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:07:49.767788986 (2h16m5.548722368s) [upload=2083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:07:49.798756949 (2h16m5.579690327s) [seg_pos={0 0}, upload=2083]: Begin upload segment... Jul 27 00:07:49.798814689 (2h16m5.579748074s) [seg_pos={0 0}, upload=2083]: Done begin upload segment. Jul 27 00:07:49.798842192 (2h16m5.57977556s) [noise=true , address=77.90.31.18:4806 , piece_id=H4UA5JHVM23TDJUZ5DR4WW26EIZVTDAT3RVABVWEMZYIUUR2OQPQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.799356728 (2h16m5.580290114s) [noise=true , address=104.36.82.16:4294 , piece_id=YWMX6IILJBNO3E2R2OHCWX6YOKJUDECASJU3LEOMQWCPDBCKRZRQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.799709141 (2h16m5.580642522s) [noise=true , address=45.91.116.5:4297 , piece_id=6V5KPO2RD74DPAVUEBJVPKLCVGMXUDMOWHHQR63MEMD5INUFZUTQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.800006755 (2h16m5.580940131s) [noise=true , address=104.36.82.10:4077 , piece_id=OTZ6FQJRXFS3GWQIC2OZKBS4GR56LPUISPEDHWWQP7EONXTK4RZQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.800439748 (2h16m5.58137313s) [noise=true , address=104.36.82.13:4186 , piece_id=3RGHTNMVOVQEMPZK4BBH3ZVAVMPJTINCMCFCM2VRSQNQDKLG6YAA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.800720006 (2h16m5.581653384s) [noise=true , address=77.90.31.18:4807 , piece_id=MJJ22ZI6ZTFOTZK67DHJ2OQSQN63QU4IKLT4DYI3PQULB36HZYPA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.801015426 (2h16m5.581948804s) [noise=true , address=77.90.31.12:4589 , piece_id=HZQC3IPR2XYKIGQXJNB3TSZPZAZAHCI7UGFPTUIXFIREGB34WPWQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.801289990 (2h16m5.582223366s) [noise=true , address=77.90.31.14:4661 , piece_id=EZEFYOBG3JCFOEICV5MFCGBVLJBJDIABGJDMBNHMMPUJDEYKHM3Q, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.801649561 (2h16m5.582582939s) [noise=true , address=77.90.31.19:4841 , piece_id=YBYPNCG5SSNEPAIAMRJANLLCERMEAVI33GBOZO566U3DZZUI3D4A, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.801972652 (2h16m5.58290603s) [noise=true , address=77.90.31.15:4699 , piece_id=LFCQEJG76VOFKXE2ZCGRXL7W2VNJL4TU4JYZU57F4DXJ55M7AYTQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.802262026 (2h16m5.583195408s) [noise=true , address=104.36.82.6:4875 , piece_id=NMBAHTMLNMDCJYB3TBNT2SEAJRJBN37UMURMD7IV7IARPNPU2SZA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.802658158 (2h16m5.583591537s) [noise=true , address=104.36.82.7:4912 , piece_id=FGKK7XKYCTCHZH47Y7EKEVBUGMLGVRBNBIWWQGMR6BPB5QIXKM2Q, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.803108676 (2h16m5.584042064s) [noise=true , address=45.91.116.12:4225 , piece_id=PE62HL7CFXWLHTIXWJ3PJJK5WSNKHWS7HDZUGVS3BV2FDCSOFTCA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.803563260 (2h16m5.584496643s) [noise=true , address=45.91.116.8:4048 , piece_id=PNZ7UI7N4R3IAWH47QAFIOWISWQDDZCLU7K5Y7DHS6OU7OX36EHA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.803878613 (2h16m5.584811998s) [noise=true , address=45.91.116.10:4083 , piece_id=ZG3MJCKUA6EKYJHDDA4AFB3D3IS6JKBKM775YUUTBIN6DLBVC4XA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.804211708 (2h16m5.585145086s) [noise=true , address=45.91.116.6:4974 , piece_id=DDGF7EDB5SBJ4H6YIYHYRFPEQ4ELLU6N272ZOKPTMF6VFRCDPVZA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.888520700 (2h16m5.669454068s) [upload=2083]: Next returned nil segment Jul 27 00:07:49.888540905 (2h16m5.669474287s) [upload=2083]: Waiting for error group managing segments... Jul 27 00:07:49.892980242 (2h16m5.673913611s) [noise=true , address=45.91.116.6:4974 , piece_id=DDGF7EDB5SBJ4H6YIYHYRFPEQ4ELLU6N272ZOKPTMF6VFRCDPVZA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:49.893437036 (2h16m5.674370404s) [noise=true , address=45.91.116.12:4225 , piece_id=PE62HL7CFXWLHTIXWJ3PJJK5WSNKHWS7HDZUGVS3BV2FDCSOFTCA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:49.893674407 (2h16m5.674607776s) [noise=true , address=77.90.31.14:4661 , piece_id=EZEFYOBG3JCFOEICV5MFCGBVLJBJDIABGJDMBNHMMPUJDEYKHM3Q, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:49.893865011 (2h16m5.674798375s) [noise=true , address=45.91.116.8:4048 , piece_id=PNZ7UI7N4R3IAWH47QAFIOWISWQDDZCLU7K5Y7DHS6OU7OX36EHA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:49.894269534 (2h16m5.675202896s) [noise=true , address=45.91.116.5:4297 , piece_id=6V5KPO2RD74DPAVUEBJVPKLCVGMXUDMOWHHQR63MEMD5INUFZUTQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:49.941852294 (2h16m5.722785663s) [noise=true , address=45.91.116.10:4083 , piece_id=ZG3MJCKUA6EKYJHDDA4AFB3D3IS6JKBKM775YUUTBIN6DLBVC4XA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:51.292263230 (2h16m7.07319661s) [noise=true , address=77.90.31.19:4841 , piece_id=YBYPNCG5SSNEPAIAMRJANLLCERMEAVI33GBOZO566U3DZZUI3D4A, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:51.299811565 (2h16m7.080744926s) [noise=true , address=77.90.31.18:4806 , piece_id=H4UA5JHVM23TDJUZ5DR4WW26EIZVTDAT3RVABVWEMZYIUUR2OQPQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:51.311794906 (2h16m7.092728283s) [noise=true , address=77.90.31.18:4807 , piece_id=MJJ22ZI6ZTFOTZK67DHJ2OQSQN63QU4IKLT4DYI3PQULB36HZYPA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:51.322780770 (2h16m7.103714378s) [noise=true , address=77.90.31.12:4589 , piece_id=HZQC3IPR2XYKIGQXJNB3TSZPZAZAHCI7UGFPTUIXFIREGB34WPWQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:08:19.739874014 (2h16m35.520807413s) [noise=true , address=104.36.82.6:4875 , piece_id=NMBAHTMLNMDCJYB3TBNT2SEAJRJBN37UMURMD7IV7IARPNPU2SZA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740039031 (2h16m35.52097241s) [noise=true , address=104.36.82.13:4186 , piece_id=3RGHTNMVOVQEMPZK4BBH3ZVAVMPJTINCMCFCM2VRSQNQDKLG6YAA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740276156 (2h16m35.521209536s) [noise=true , address=104.36.82.16:4294 , piece_id=YWMX6IILJBNO3E2R2OHCWX6YOKJUDECASJU3LEOMQWCPDBCKRZRQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740653885 (2h16m35.52158727s) [noise=true , address=104.36.82.7:4912 , piece_id=FGKK7XKYCTCHZH47Y7EKEVBUGMLGVRBNBIWWQGMR6BPB5QIXKM2Q, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740760999 (2h16m35.521694375s) [noise=true , address=104.36.82.10:4077 , piece_id=OTZ6FQJRXFS3GWQIC2OZKBS4GR56LPUISPEDHWWQP7EONXTK4RZQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740998689 (2h16m35.521932116s) [noise=true , address=77.90.31.15:4699 , piece_id=LFCQEJG76VOFKXE2ZCGRXL7W2VNJL4TU4JYZU57F4DXJ55M7AYTQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.741048351 (2h16m35.52198174s) [seg_pos={0 0}, upload=2083]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:08:19.764313990 (2h16m35.545247374s) [upload=2083]: Done uploading segments... Jul 27 00:08:19.764346822 (2h16m35.545280207s) [upload=2083]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.027078164s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XXX...XX.....X.XXX...............X..X......X.XX....X...X.] duration: 1.463660392s FAIL (2025-07-27 00:08:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.463683774s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XX] duration: 30.029461798s FAIL (2025-07-27 00:08:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.029491526s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029778088s FAIL (2025-07-27 00:07:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=2068]: Starting upload Jul 27 00:06:58.373532907 (2h15m14.154466276s) [upload=2068]: Done starting upload Jul 27 00:06:58.373651073 (2h15m14.154584441s) [upload=2068]: Uploading segments... Jul 27 00:06:58.373937665 (2h15m14.154871036s) [upload=2068]: Got next segment. Inline: false Jul 27 00:06:58.373984986 (2h15m14.154918354s) [upload=2068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:06:58.373991665 (2h15m14.154925032s) [upload=2068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:06:58.413242534 (2h15m14.19417591s) [seg_pos={0 0}, upload=2068]: Begin upload segment... Jul 27 00:06:58.413320530 (2h15m14.194253909s) [noise=true , address=79.127.219.38:4010 , piece_id=PVX626CPFPWPP5Z34RLZF6HT4LBFEHKDIFINFGUW7QLVNIUK462Q, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.413934784 (2h15m14.194868173s) [noise=true , address=79.127.205.245:4240 , piece_id=NW7QFPDHNP3XDREHQZDGVGOWBL4NSRPF432QNKNZHJ7FRBLLEDOA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.414450868 (2h15m14.195384252s) [noise=true , address=64.49.13.138:4488 , piece_id=WPNNCSMDIJQJK47Z7TGWXFM35NZL4STMCF33EV6PCCCPBE7W25QQ, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.414798904 (2h15m14.195732291s) [seg_pos={0 0}, upload=2068]: Done begin upload segment. Jul 27 00:06:58.414830152 (2h15m14.195763547s) [noise=true , address=104.36.82.47:4522 , piece_id=EBH2HJFBR5KRPGLXWP4RXMHVYURRVLITFXHDSY6WBBJHDCJZNFOQ, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.415225560 (2h15m14.196158942s) [noise=true , address=77.90.31.11:4573 , piece_id=ZQROWPUM5TX76GFAT2NMHT4SVJ4N25AMOBPO4ZBNZ4EEI2Z3RM7A, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.415951604 (2h15m14.19688499s) [noise=true , address=79.127.219.51:4210 , piece_id=QAL4UEWTVB3TG2G2PP33C3REG4X32PYNG5GMZX5FZBYWRIVM5TEA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.416156711 (2h15m14.197090103s) [noise=true , address=79.127.219.48:4074 , piece_id=77WFCSA62AP3YRQZYUAU4NEQ2VO32OLDUCK22OZYHKWMG65AAPNQ, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.416834350 (2h15m14.197767736s) [noise=true , address=79.127.205.234:4610 , piece_id=5UKXEIQUIJ36ESPZGUEYJYBOHX6KJ4J6TWZBYJ6NTDTG2SGLQKRA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.417407625 (2h15m14.198340993s) [noise=true , address=104.36.82.53:4138 , piece_id=BYFFJJH7OWDW3RL2BLNF4JJJDDIJNKZ7N3QAS5DKBUKYEVFRFTXA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.418647613 (2h15m14.199581s) [noise=true , address=64.49.13.137:4502 , piece_id=XHI442SNUIFHZEZB367WTQRCDPFMJ6YOG77QNOK5BADS4FMFQH4A, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.419720953 (2h15m14.20065434s) [noise=true , address=79.127.219.47:4048 , piece_id=I3ZMDYNVLI2NXDM73TZOHCL3NCJEJEVQYFGY7RFZURSC4O4QVT3A, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.420409375 (2h15m14.201342756s) [noise=true , address=104.36.82.54:4175 , piece_id=CYTWMWDQKJOWD2DWMYZ64F66RIIN7ANKVPL5EZUFTXRQF4UTDBLA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.421117273 (2h15m14.202050661s) [noise=true , address=45.91.116.9:4277 , piece_id=B6UDYRSHLLNGA3M22QURXGQ7TYHZY7ER6VI5XPO33QADHP5ZZP5Q, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.421667155 (2h15m14.202600539s) [noise=true , address=79.127.205.233:4655 , piece_id=2CHZN3DRC4MH3ZWW4DDC5EWAQQ7XSPL5ALWJFEEDLGFRNANDNIOA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.421730342 (2h15m14.202663705s) [noise=true , address=104.36.82.21:4497 , piece_id=5C2ZJDDTAVAORHXHE4J4SGGOQL7F55RY426Z7SMJTM5PNXWRQ67A, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.423126966 (2h15m14.204060354s) [noise=true , address=104.36.82.46:4496 , piece_id=QYXV7PQLHVMDT5GH3OTPHBYW3NCK3WNJSQT7ZVGXNBUUXABFSINA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.423236450 (2h15m14.204169813s) [noise=true , address=77.90.31.16:4736 , piece_id=YQF5BNYVS4ECPTNZ6VQ47KXR24X5ZDO2KBEU6LCJYRQPADRSJKMQ, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.424063221 (2h15m14.204996584s) [noise=true , address=79.127.201.215:4360 , piece_id=ZFD67RCH337WY4YZEFNOLELSQUXHFVPWE33GXQ4DIG6AXRITG46A, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.508849014 (2h15m14.289782383s) [upload=2068]: Next returned nil segment Jul 27 00:06:58.508872409 (2h15m14.289805808s) [upload=2068]: Waiting for error group managing segments... Jul 27 00:06:58.513728587 (2h15m14.294661952s) [noise=true , address=79.127.219.38:4010 , piece_id=PVX626CPFPWPP5Z34RLZF6HT4LBFEHKDIFINFGUW7QLVNIUK462Q, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:58.533893760 (2h15m14.31482713s) [noise=true , address=77.90.31.16:4736 , piece_id=YQF5BNYVS4ECPTNZ6VQ47KXR24X5ZDO2KBEU6LCJYRQPADRSJKMQ, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:58.553290858 (2h15m14.334224247s) [noise=true , address=77.90.31.11:4573 , piece_id=ZQROWPUM5TX76GFAT2NMHT4SVJ4N25AMOBPO4ZBNZ4EEI2Z3RM7A, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:58.610490363 (2h15m14.391423755s) [noise=true , address=79.127.219.47:4048 , piece_id=I3ZMDYNVLI2NXDM73TZOHCL3NCJEJEVQYFGY7RFZURSC4O4QVT3A, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:59.029814337 (2h15m14.810747703s) [noise=true , address=79.127.205.245:4240 , piece_id=NW7QFPDHNP3XDREHQZDGVGOWBL4NSRPF432QNKNZHJ7FRBLLEDOA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:59.058372704 (2h15m14.839306069s) [noise=true , address=79.127.205.233:4655 , piece_id=2CHZN3DRC4MH3ZWW4DDC5EWAQQ7XSPL5ALWJFEEDLGFRNANDNIOA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:59.099820554 (2h15m14.880753937s) [noise=true , address=64.49.13.137:4502 , piece_id=XHI442SNUIFHZEZB367WTQRCDPFMJ6YOG77QNOK5BADS4FMFQH4A, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:59.425431871 (2h15m15.20636525s) [noise=true , address=79.127.205.234:4610 , piece_id=5UKXEIQUIJ36ESPZGUEYJYBOHX6KJ4J6TWZBYJ6NTDTG2SGLQKRA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:00.319004901 (2h15m16.099938282s) [noise=true , address=79.127.219.51:4210 , piece_id=QAL4UEWTVB3TG2G2PP33C3REG4X32PYNG5GMZX5FZBYWRIVM5TEA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:01.785238153 (2h15m17.566171515s) [noise=true , address=79.127.219.48:4074 , piece_id=77WFCSA62AP3YRQZYUAU4NEQ2VO32OLDUCK22OZYHKWMG65AAPNQ, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:08.239746995 (2h15m24.020680361s) [noise=true , address=64.49.13.138:4488 , piece_id=WPNNCSMDIJQJK47Z7TGWXFM35NZL4STMCF33EV6PCCCPBE7W25QQ, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:17.856709238 (2h15m33.637642602s) [noise=true , address=45.91.116.9:4277 , piece_id=B6UDYRSHLLNGA3M22QURXGQ7TYHZY7ER6VI5XPO33QADHP5ZZP5Q, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:20.148635163 (2h15m35.929568551s) [noise=true , address=79.127.201.215:4360 , piece_id=ZFD67RCH337WY4YZEFNOLELSQUXHFVPWE33GXQ4DIG6AXRITG46A, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:58.337825465 (2h16m14.11875885s) [noise=true , address=104.36.82.53:4138 , piece_id=BYFFJJH7OWDW3RL2BLNF4JJJDDIJNKZ7N3QAS5DKBUKYEVFRFTXA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: rpc: dial tcp 104.36.82.53:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:58.337936421 (2h16m14.118869798s) [noise=true , address=104.36.82.21:4497 , piece_id=5C2ZJDDTAVAORHXHE4J4SGGOQL7F55RY426Z7SMJTM5PNXWRQ67A, seg_pos={0 0}, upload=2068]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:104.36.82.21:4497): protocol: rpc: dial tcp 104.36.82.21:4497: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:58.338008663 (2h16m14.118942045s) [noise=true , address=104.36.82.47:4522 , piece_id=EBH2HJFBR5KRPGLXWP4RXMHVYURRVLITFXHDSY6WBBJHDCJZNFOQ, seg_pos={0 0}, upload=2068]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: rpc: dial tcp 104.36.82.47:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:58.338077348 (2h16m14.119010725s) [noise=true , address=104.36.82.54:4175 , piece_id=CYTWMWDQKJOWD2DWMYZ64F66RIIN7ANKVPL5EZUFTXRQF4UTDBLA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:58.338155524 (2h16m14.119088894s) [noise=true , address=104.36.82.46:4496 , piece_id=QYXV7PQLHVMDT5GH3OTPHBYW3NCK3WNJSQT7ZVGXNBUUXABFSINA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: write tcp 159.223.143.39:59222->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:58.338186479 (2h16m14.119119858s) [seg_pos={0 0}, upload=2068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 27 00:07:58.365161557 (2h16m14.146094941s) [upload=2068]: Done uploading segments... Jul 27 00:07:58.365198927 (2h16m14.146132307s) [upload=2068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.029802245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....XXX...XX.....X.XXX...............X..X......X.XX....X...X] duration: 30.004359571s FAIL (2025-07-27 00:07:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.004377516s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..X] duration: 30.026333626s FAIL (2025-07-27 00:07:33 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.026390667s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....X...X....XX..............X.X........X...XX.............] duration: 24.005517359s FAIL (2025-07-27 00:07:25 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.005541934s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [......X...X....XX..............X.X........X...XX............] duration: 52.433451111s FAIL (2025-07-27 00:07:00 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.433482642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029601401s FAIL (2025-07-27 00:06:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05:58.285655896 (2h14m14.066589275s) [upload=2055]: Got next segment. Inline: false Jul 27 00:05:58.285750294 (2h14m14.066683671s) [upload=2055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:05:58.285759073 (2h14m14.066692453s) [upload=2055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:05:58.325522388 (2h14m14.106455765s) [seg_pos={0 0}, upload=2055]: Begin upload segment... Jul 27 00:05:58.325582098 (2h14m14.106515463s) [seg_pos={0 0}, upload=2055]: Done begin upload segment. Jul 27 00:05:58.325614332 (2h14m14.106547729s) [noise=true , address=64.49.13.138:4488 , piece_id=CZBKAYLTRLOTMU2QA2Q5THD4YVFGQ6SGPFQCOCCWE6MZ55S42GZQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.325647583 (2h14m14.106580973s) [noise=true , address=104.36.82.46:4513 , piece_id=RFUY2S6J5LTKLMCPJ5XUALZKVRLTVGQUXB2N7DUUL6LNXT2LRRAQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326034066 (2h14m14.106967445s) [noise=true , address=79.127.205.245:4227 , piece_id=NMI7IN2I546RHHWPK3G6YGFII6EMDMLKC7KH5QQ6AXQ5PVXC4DIQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326248547 (2h14m14.107181933s) [noise=true , address=104.36.82.42:4807 , piece_id=QHVUNNT6BC5PUJTUF2GK3NZVUS3QHMEGHR5Q3HSKY33GCPU3XXFQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326353861 (2h14m14.107287241s) [noise=true , address=79.127.205.233:4624 , piece_id=MM2AMS7F65ACLP7K7T4GPSLWSCZRAFYPSEAEICMVBIH27WPQUAYA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326652705 (2h14m14.107586085s) [noise=true , address=104.36.82.47:4518 , piece_id=GYTLS7JW2CE67UI6XMAZBHSFDGO4X2N7PHBVRFHUNJ6AWXRAG33A, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326757970 (2h14m14.107691357s) [noise=true , address=104.36.82.56:4263 , piece_id=FX7AFYY7VA6LXBQXQBB6K6QFVDM7SSOL2M4HJW7ZZS4JNKODWEYQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326946463 (2h14m14.107879839s) [noise=true , address=104.36.82.41:4792 , piece_id=3VOW6ULEXAMKY5XKUPN2M2OF5QOC7UKWLNO5MGCQRQCLXOKL7DIQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.327291335 (2h14m14.108224719s) [noise=true , address=79.127.205.235:4678 , piece_id=ROLUJUTRCRCPUCH2UDR6CNL75TGR6OK4VP5DLJ5YZBG3SMBVYOIA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.327302756 (2h14m14.108236132s) [noise=true , address=77.90.31.12:4599 , piece_id=VUO6TD6RHSC2AS4W35SJE76FVPHDUAJSFIKYU3O4PKOD3A2BEEQA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.327622421 (2h14m14.108555804s) [noise=true , address=79.127.220.101:4041 , piece_id=SNPFNKV2APLHHCZOO6NZFJ5GYLSDHA6PXZ7Q4OY7VSOFLJMLP5MA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.327835159 (2h14m14.108768535s) [noise=true , address=79.127.220.101:4048 , piece_id=5JCBUFMQ26LZO3V63U7TDF6EIC3CVMNLH2F4CXAWIJEP2X3CBTNA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.327999948 (2h14m14.108933328s) [noise=true , address=77.90.31.13:4629 , piece_id=YCL43MLCDT7H5JEXZTJPFZPBYPCQDU5OBCZUDF6BWVCL6WH7D3WA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.328534828 (2h14m14.10946821s) [noise=true , address=79.127.205.246:4284 , piece_id=MOZVL2IO5BVCGGXMCEHBR44N75Q6X4UFGKYGX7B5PJHBIRSV3PIA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.329101753 (2h14m14.110035139s) [noise=true , address=77.90.31.11:4581 , piece_id=L5FL6RZAKVEEVE3FKYVSKMMWSZUE4AGCSIDPIPBUVNXOJ4N2PTMA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.330680714 (2h14m14.111614101s) [noise=true , address=79.127.201.215:4351 , piece_id=HWJEB3QRKBAYY77XPW43B5QLIGZO7IKROYY7BRLBJZ2WJI2UR62A, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.330787939 (2h14m14.111721323s) [noise=true , address=79.127.205.234:4607 , piece_id=IBB23QLHILMWRMWRCF4525OXXHOAD5F2WBGACMEQFSAU2RQKKMLQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.332921640 (2h14m14.113855013s) [noise=true , address=77.90.31.21:4896 , piece_id=C4KLG3F3ZFKGDLCVQ5TYMLLDY435RYGFKH3MRBMADOHP7ZK33LXA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.376280609 (2h14m14.157213996s) [noise=true , address=77.90.31.13:4629 , piece_id=YCL43MLCDT7H5JEXZTJPFZPBYPCQDU5OBCZUDF6BWVCL6WH7D3WA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: storage node overloaded, request limit: 2000 Jul 27 00:05:58.421789149 (2h14m14.20272253s) [upload=2055]: Next returned nil segment Jul 27 00:05:58.421812577 (2h14m14.202745964s) [upload=2055]: Waiting for error group managing segments... Jul 27 00:05:58.475786404 (2h14m14.256719782s) [noise=true , address=77.90.31.21:4896 , piece_id=C4KLG3F3ZFKGDLCVQ5TYMLLDY435RYGFKH3MRBMADOHP7ZK33LXA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.514804015 (2h14m14.295737384s) [noise=true , address=77.90.31.11:4581 , piece_id=L5FL6RZAKVEEVE3FKYVSKMMWSZUE4AGCSIDPIPBUVNXOJ4N2PTMA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.909211035 (2h14m14.690144414s) [noise=true , address=79.127.205.234:4607 , piece_id=IBB23QLHILMWRMWRCF4525OXXHOAD5F2WBGACMEQFSAU2RQKKMLQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.927170040 (2h14m14.708103421s) [noise=true , address=79.127.205.246:4284 , piece_id=MOZVL2IO5BVCGGXMCEHBR44N75Q6X4UFGKYGX7B5PJHBIRSV3PIA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.934110828 (2h14m14.715044202s) [noise=true , address=79.127.205.233:4624 , piece_id=MM2AMS7F65ACLP7K7T4GPSLWSCZRAFYPSEAEICMVBIH27WPQUAYA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.955956305 (2h14m14.736889685s) [noise=true , address=79.127.205.235:4678 , piece_id=ROLUJUTRCRCPUCH2UDR6CNL75TGR6OK4VP5DLJ5YZBG3SMBVYOIA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.999211680 (2h14m14.780145047s) [noise=true , address=79.127.205.245:4227 , piece_id=NMI7IN2I546RHHWPK3G6YGFII6EMDMLKC7KH5QQ6AXQ5PVXC4DIQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:59.158673902 (2h14m14.939607287s) [noise=true , address=79.127.220.101:4048 , piece_id=5JCBUFMQ26LZO3V63U7TDF6EIC3CVMNLH2F4CXAWIJEP2X3CBTNA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:06:01.209634590 (2h14m16.990567973s) [noise=true , address=64.49.13.138:4488 , piece_id=CZBKAYLTRLOTMU2QA2Q5THD4YVFGQ6SGPFQCOCCWE6MZ55S42GZQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:06:01.966716465 (2h14m17.747649837s) [noise=true , address=79.127.220.101:4041 , piece_id=SNPFNKV2APLHHCZOO6NZFJ5GYLSDHA6PXZ7Q4OY7VSOFLJMLP5MA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:06:13.932563845 (2h14m29.71349723s) [noise=true , address=79.127.201.215:4351 , piece_id=HWJEB3QRKBAYY77XPW43B5QLIGZO7IKROYY7BRLBJZ2WJI2UR62A, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:06:26.446548545 (2h14m42.227481913s) [noise=true , address=77.90.31.12:4599 , piece_id=VUO6TD6RHSC2AS4W35SJE76FVPHDUAJSFIKYU3O4PKOD3A2BEEQA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:06:58.246628477 (2h15m14.027561863s) [noise=true , address=104.36.82.42:4807 , piece_id=QHVUNNT6BC5PUJTUF2GK3NZVUS3QHMEGHR5Q3HSKY33GCPU3XXFQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:1iTiX3X69T9rKBxe1MLwpJEJPZyXVATQBP9cDEnQe2riXMGZeZ, address:104.36.82.42:4807): protocol: rpc: dial tcp 104.36.82.42:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:58.246730364 (2h15m14.02766375s) [noise=true , address=104.36.82.56:4263 , piece_id=FX7AFYY7VA6LXBQXQBB6K6QFVDM7SSOL2M4HJW7ZZS4JNKODWEYQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:12tLW4r8AQKLCbSWZoNUVi5v2YtF8NthUmX81TLQn8G8TotuByG, address:104.36.82.56:4263): protocol: rpc: dial tcp 104.36.82.56:4263: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:58.246918178 (2h15m14.027851559s) [noise=true , address=104.36.82.41:4792 , piece_id=3VOW6ULEXAMKY5XKUPN2M2OF5QOC7UKWLNO5MGCQRQCLXOKL7DIQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: rpc: dial tcp 104.36.82.41:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:58.246931972 (2h15m14.027865357s) [noise=true , address=104.36.82.46:4513 , piece_id=RFUY2S6J5LTKLMCPJ5XUALZKVRLTVGQUXB2N7DUUL6LNXT2LRRAQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 159.223.143.39:60616->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:58.246983250 (2h15m14.027916638s) [noise=true , address=104.36.82.47:4518 , piece_id=GYTLS7JW2CE67UI6XMAZBHSFDGO4X2N7PHBVRFHUNJ6AWXRAG33A, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:104.36.82.47:4518): protocol: write tcp 159.223.143.39:42512->104.36.82.47:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:58.247215080 (2h15m14.028148494s) [seg_pos={0 0}, upload=2055]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:06:58.272223521 (2h15m14.053156899s) [upload=2055]: Done uploading segments... Jul 27 00:06:58.272255127 (2h15m14.053188508s) [upload=2055]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.029620968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....] duration: 1.058837634s FAIL (2025-07-27 00:06:39 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:56.936460143 (2h11m12.717393528s) [upload=2012]: Starting upload Jul 27 00:02:56.936557479 (2h11m12.717490863s) [upload=2012]: Done starting upload Jul 27 00:02:56.936647830 (2h11m12.717581213s) [upload=2012]: Uploading segments... Jul 27 00:02:56.937017175 (2h11m12.71795055s) [upload=2012]: Got next segment. Inline: false Jul 27 00:02:56.937075780 (2h11m12.71800916s) [upload=2012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:56.937082791 (2h11m12.71801617s) [upload=2012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:56.970415476 (2h11m12.751348861s) [seg_pos={0 0}, upload=2012]: Begin upload segment... Jul 27 00:02:56.970488660 (2h11m12.751422026s) [seg_pos={0 0}, upload=2012]: Done begin upload segment. Jul 27 00:02:56.970499199 (2h11m12.751432579s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970507686 (2h11m12.751441073s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970999895 (2h11m12.75193328s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.971544420 (2h11m12.752477804s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974009791 (2h11m12.754943174s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974480217 (2h11m12.755413597s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974879774 (2h11m12.755813153s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975304158 (2h11m12.756237538s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975704939 (2h11m12.756638317s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976240365 (2h11m12.757173746s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976662963 (2h11m12.757596345s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977149854 (2h11m12.75808323s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977565879 (2h11m12.758499264s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977962142 (2h11m12.758895525s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978370394 (2h11m12.759303791s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978686570 (2h11m12.759619953s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:57.069515194 (2h11m12.850448562s) [upload=2012]: Next returned nil segment Jul 27 00:02:57.069531895 (2h11m12.850465271s) [upload=2012]: Waiting for error group managing segments... Jul 27 00:02:57.073615634 (2h11m12.854549001s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074286917 (2h11m12.855220285s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074641770 (2h11m12.855575146s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074773340 (2h11m12.855706716s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075293141 (2h11m12.856226509s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075311903 (2h11m12.856245271s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075673862 (2h11m12.856607227s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075747686 (2h11m12.856681054s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.139972458 (2h11m12.920905828s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.172759261 (2h11m12.953692623s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:58.458246223 (2h11m14.2391796s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:03:26.903729789 (2h11m42.684663172s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903813623 (2h11m42.684747001s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903844203 (2h11m42.684777573s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903891506 (2h11m42.684824887s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:35200->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903946088 (2h11m42.684879465s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903986605 (2h11m42.684919991s) [seg_pos={0 0}, upload=2012]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:03:26.932638927 (2h11m42.713572307s) [upload=2012]: Done uploading segments... Jul 27 00:03:26.932697222 (2h11m42.713630598s) [upload=2012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.058867959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..] duration: 484.575542ms FAIL (2025-07-27 00:06:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 484.609457ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.......X...X....XX..............X.X........X...XX...........] duration: 42.776574641s FAIL (2025-07-27 00:06:08 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.776642484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....XXX...XX.....X.XXX...............X..X......X.XX....X...] duration: 2.069217237s FAIL (2025-07-27 00:06:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.069252801s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.037864554s FAIL (2025-07-27 00:05:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== SegmentBegin Jul 27 00:04:58.210205117 (2h13m13.991138509s) [seg_pos={0 0}, upload=2040]: Begin upload segment... Jul 27 00:04:58.210287779 (2h13m13.991221171s) [seg_pos={0 0}, upload=2040]: Done begin upload segment. Jul 27 00:04:58.210650742 (2h13m13.99158413s) [noise=true , address=104.36.82.41:4780 , piece_id=2SVTYDMLGEELWC4XH6TLASYMUMW6LXX4QIINTY4PI7MSJ6SW26FQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.210711134 (2h13m13.991644513s) [noise=true , address=79.127.205.245:4226 , piece_id=DTHBLIXNKHVZUFKIGO4KV7AMKYEE5GFT4CK7P7E7HID2D4SYJZGA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.211869232 (2h13m13.992802622s) [noise=true , address=79.127.205.234:4603 , piece_id=BYY3L5USEL43NCLA3KBVXL5S4X6RKL3FPUAWRTZC5K5OINIJRDYQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.212969363 (2h13m13.993902755s) [noise=true , address=104.36.82.54:4166 , piece_id=OJEMT4YGML5DH2QAU4UWPTXG3BEA2PZBOKZTB5V3WPR5SK3SNLQA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.213491372 (2h13m13.994424759s) [noise=true , address=79.127.201.215:4349 , piece_id=7M5Y5INLLV3UMNBDRRKW737IJE546BQTBJ2FXBRXPGEKXJ44ZOQQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.213698188 (2h13m13.994631581s) [noise=true , address=64.49.13.140:4508 , piece_id=OOMWZTH5VHSX26YNTY2OIAD4DNVYM4WVCIPDE2EYLUB4GPU3CPUA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.213916501 (2h13m13.994849888s) [noise=true , address=104.36.82.6:4887 , piece_id=LHGUXVTS3VNN4UJAZN5ED7GUACCA4OFRHXTSEOEZGP3LJ5IMP5RA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.214398346 (2h13m13.995331734s) [noise=true , address=104.36.82.47:4526 , piece_id=AWELNJMDXZU2LG35FEOU6DLK2ZH6KQBPOQYLUPKYGFPA54QR2TWA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.214430650 (2h13m13.995364022s) [noise=true , address=79.127.201.214:4289 , piece_id=AYEEO6F6PDE3BZQZU5PCERK65AQ27SR5NCEHXDY2RENI5YSIVWUQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.214900805 (2h13m13.99583419s) [noise=true , address=79.127.219.38:4004 , piece_id=NLFZ7OY7KOIQJUGFV27SNXDENPQJZXQW3HLE2S26SBZX7RS72NVQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.215513764 (2h13m13.996447153s) [noise=true , address=45.91.116.10:4093 , piece_id=DC22SLYAVXFR7N5RAAWXBZ3MY4TIYSSLCYLF3YOUWUC6V55XB6WA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.215962482 (2h13m13.996895861s) [noise=true , address=104.36.82.8:4001 , piece_id=4ORP7YRXKVDRBY6XSLWFWGZZ7E7KGO6LU3APKQXXIUBHMBHSHTVQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.216023073 (2h13m13.996956464s) [noise=true , address=79.127.205.246:4288 , piece_id=DDSLWTUZX26JF6OJJMRU63AW5ZYJJNNBV4HBUESAT7TO4IN27LSA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.216651063 (2h13m13.997584452s) [noise=true , address=79.127.205.235:4675 , piece_id=3DQRQ3LPZLD32VEAYWJS2K7KVG53M3O2OXHXLXKB423Q4X6WP7NA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.217179857 (2h13m13.998113247s) [noise=true , address=77.90.31.21:4880 , piece_id=SK3YBP32VTDVNJMRWPN3A5JLWVG5OMHUI54KRJ5FDEGVYMCX6MYA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.218088634 (2h13m13.999022026s) [noise=true , address=79.127.219.47:4048 , piece_id=DNKSVGGE2TYRPIVA5TE2BW5PG2S64UW7XS6BLLAJD6KGX7GD63EQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.218655413 (2h13m13.99958879s) [noise=true , address=77.90.31.13:4636 , piece_id=SAZB4S7UXMME7YIHVAABYYMLJYZFUWDXYKXXIFP7JLZGU2B5H22Q, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.218694842 (2h13m13.999628229s) [noise=true , address=104.36.82.52:4094 , piece_id=TOGFZPIGG7JSRP2ZTGDDZ7KC3HJSM22ERNPN6JDOH7NRBIULRSDQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.315897196 (2h13m14.096830584s) [upload=2040]: Next returned nil segment Jul 27 00:04:58.315922828 (2h13m14.096856216s) [upload=2040]: Waiting for error group managing segments... Jul 27 00:04:58.344839771 (2h13m14.125773148s) [noise=true , address=79.127.219.38:4004 , piece_id=NLFZ7OY7KOIQJUGFV27SNXDENPQJZXQW3HLE2S26SBZX7RS72NVQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.361300725 (2h13m14.142234087s) [noise=true , address=77.90.31.21:4880 , piece_id=SK3YBP32VTDVNJMRWPN3A5JLWVG5OMHUI54KRJ5FDEGVYMCX6MYA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.808295224 (2h13m14.589228599s) [noise=true , address=79.127.201.215:4349 , piece_id=7M5Y5INLLV3UMNBDRRKW737IJE546BQTBJ2FXBRXPGEKXJ44ZOQQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.811673220 (2h13m14.592606595s) [noise=true , address=79.127.205.245:4226 , piece_id=DTHBLIXNKHVZUFKIGO4KV7AMKYEE5GFT4CK7P7E7HID2D4SYJZGA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.845410835 (2h13m14.626344204s) [noise=true , address=79.127.205.234:4603 , piece_id=BYY3L5USEL43NCLA3KBVXL5S4X6RKL3FPUAWRTZC5K5OINIJRDYQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.868675076 (2h13m14.649608445s) [noise=true , address=79.127.205.246:4288 , piece_id=DDSLWTUZX26JF6OJJMRU63AW5ZYJJNNBV4HBUESAT7TO4IN27LSA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.869733797 (2h13m14.650667167s) [noise=true , address=79.127.205.235:4675 , piece_id=3DQRQ3LPZLD32VEAYWJS2K7KVG53M3O2OXHXLXKB423Q4X6WP7NA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:59.073906603 (2h13m14.85483998s) [noise=true , address=79.127.219.47:4048 , piece_id=DNKSVGGE2TYRPIVA5TE2BW5PG2S64UW7XS6BLLAJD6KGX7GD63EQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:05:01.603186112 (2h13m17.384119495s) [noise=true , address=79.127.201.214:4289 , piece_id=AYEEO6F6PDE3BZQZU5PCERK65AQ27SR5NCEHXDY2RENI5YSIVWUQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:05:01.656202627 (2h13m17.437136008s) [noise=true , address=77.90.31.13:4636 , piece_id=SAZB4S7UXMME7YIHVAABYYMLJYZFUWDXYKXXIFP7JLZGU2B5H22Q, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:05:07.987842829 (2h13m23.768776207s) [noise=true , address=45.91.116.10:4093 , piece_id=DC22SLYAVXFR7N5RAAWXBZ3MY4TIYSSLCYLF3YOUWUC6V55XB6WA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:05:09.863488221 (2h13m25.644421585s) [noise=true , address=64.49.13.140:4508 , piece_id=OOMWZTH5VHSX26YNTY2OIAD4DNVYM4WVCIPDE2EYLUB4GPU3CPUA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:05:58.136527840 (2h14m13.917461217s) [noise=true , address=104.36.82.8:4001 , piece_id=4ORP7YRXKVDRBY6XSLWFWGZZ7E7KGO6LU3APKQXXIUBHMBHSHTVQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:12Eaef5RrMHiZS6dE7wTXyq239F1RXovU7UCTQ7uC6Qusj6bJDw, address:104.36.82.8:4001): protocol: rpc: dial tcp 104.36.82.8:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.136601036 (2h14m13.917534405s) [noise=true , address=104.36.82.41:4780 , piece_id=2SVTYDMLGEELWC4XH6TLASYMUMW6LXX4QIINTY4PI7MSJ6SW26FQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:104.36.82.41:4780): protocol: write tcp 159.223.143.39:55260->104.36.82.41:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.136647311 (2h14m13.917580676s) [noise=true , address=104.36.82.6:4887 , piece_id=LHGUXVTS3VNN4UJAZN5ED7GUACCA4OFRHXTSEOEZGP3LJ5IMP5RA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7, address:104.36.82.6:4887): protocol: write tcp 159.223.143.39:38140->104.36.82.6:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.136796710 (2h14m13.917730089s) [noise=true , address=104.36.82.54:4166 , piece_id=OJEMT4YGML5DH2QAU4UWPTXG3BEA2PZBOKZTB5V3WPR5SK3SNLQA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: write tcp 159.223.143.39:40486->104.36.82.54:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.136914293 (2h14m13.917847658s) [noise=true , address=104.36.82.47:4526 , piece_id=AWELNJMDXZU2LG35FEOU6DLK2ZH6KQBPOQYLUPKYGFPA54QR2TWA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: rpc: dial tcp 104.36.82.47:4526: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.136970440 (2h14m13.917903816s) [noise=true , address=104.36.82.52:4094 , piece_id=TOGFZPIGG7JSRP2ZTGDDZ7KC3HJSM22ERNPN6JDOH7NRBIULRSDQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: write tcp 159.223.143.39:33056->104.36.82.52:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.137019619 (2h14m13.917953002s) [seg_pos={0 0}, upload=2040]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:05:58.171599389 (2h14m13.952532762s) [upload=2040]: Done uploading segments... Jul 27 00:05:58.171630335 (2h14m13.952563721s) [upload=2040]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.037888634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX...] duration: 507.313538ms FAIL (2025-07-27 00:05:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:56.936460143 (2h11m12.717393528s) [upload=2012]: Starting upload Jul 27 00:02:56.936557479 (2h11m12.717490863s) [upload=2012]: Done starting upload Jul 27 00:02:56.936647830 (2h11m12.717581213s) [upload=2012]: Uploading segments... Jul 27 00:02:56.937017175 (2h11m12.71795055s) [upload=2012]: Got next segment. Inline: false Jul 27 00:02:56.937075780 (2h11m12.71800916s) [upload=2012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:56.937082791 (2h11m12.71801617s) [upload=2012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:56.970415476 (2h11m12.751348861s) [seg_pos={0 0}, upload=2012]: Begin upload segment... Jul 27 00:02:56.970488660 (2h11m12.751422026s) [seg_pos={0 0}, upload=2012]: Done begin upload segment. Jul 27 00:02:56.970499199 (2h11m12.751432579s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970507686 (2h11m12.751441073s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970999895 (2h11m12.75193328s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.971544420 (2h11m12.752477804s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974009791 (2h11m12.754943174s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974480217 (2h11m12.755413597s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974879774 (2h11m12.755813153s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975304158 (2h11m12.756237538s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975704939 (2h11m12.756638317s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976240365 (2h11m12.757173746s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976662963 (2h11m12.757596345s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977149854 (2h11m12.75808323s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977565879 (2h11m12.758499264s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977962142 (2h11m12.758895525s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978370394 (2h11m12.759303791s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978686570 (2h11m12.759619953s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:57.069515194 (2h11m12.850448562s) [upload=2012]: Next returned nil segment Jul 27 00:02:57.069531895 (2h11m12.850465271s) [upload=2012]: Waiting for error group managing segments... Jul 27 00:02:57.073615634 (2h11m12.854549001s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074286917 (2h11m12.855220285s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074641770 (2h11m12.855575146s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074773340 (2h11m12.855706716s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075293141 (2h11m12.856226509s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075311903 (2h11m12.856245271s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075673862 (2h11m12.856607227s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075747686 (2h11m12.856681054s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.139972458 (2h11m12.920905828s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.172759261 (2h11m12.953692623s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:58.458246223 (2h11m14.2391796s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:03:26.903729789 (2h11m42.684663172s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903813623 (2h11m42.684747001s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903844203 (2h11m42.684777573s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903891506 (2h11m42.684824887s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:35200->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903946088 (2h11m42.684879465s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903986605 (2h11m42.684919991s) [seg_pos={0 0}, upload=2012]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:03:26.932638927 (2h11m42.713572307s) [upload=2012]: Done uploading segments... Jul 27 00:03:26.932697222 (2h11m42.713630598s) [upload=2012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 507.343223ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX.] duration: 1.809832875s FAIL (2025-07-27 00:05:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.809865677s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX] duration: 30.061581937s FAIL (2025-07-27 00:05:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.061609324s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXX] duration: 30.029981076s FAIL (2025-07-27 00:05:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.030025148s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027872642s FAIL (2025-07-27 00:04:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 256 (2h12m13.851182632s) [upload=2025]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:03:58.114046876 (2h12m13.894980387s) [seg_pos={0 0}, upload=2025]: Begin upload segment... Jul 27 00:03:58.114113103 (2h12m13.895046484s) [seg_pos={0 0}, upload=2025]: Done begin upload segment. Jul 27 00:03:58.114152307 (2h12m13.895085693s) [noise=true , address=64.49.13.154:4927 , piece_id=3M4S26SJQTYS5WYS5B6KF6LGTYBXRHS2J6AHI26BOP4K5M22YXLQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.129874143 (2h12m13.910807531s) [noise=true , address=104.36.82.54:4170 , piece_id=BVFZN6P2XD6AEA34MZKT4MN7PPGVUHUEDT6RQFVPOIMHCL4NUGZQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.130394360 (2h12m13.911327748s) [noise=true , address=77.90.31.13:4636 , piece_id=EMBHHFZ3NKBGCO5E77DTO7QFR73CHRWLOYHZRAJJNRJSXYJBNIWA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.130997070 (2h12m13.911930457s) [noise=true , address=104.36.82.47:4549 , piece_id=6UDTTFYCYC5LBH5G4WM54UU6YRG2U4B65LONOHMOK3VIPKUTM7PQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.131588284 (2h12m13.912521671s) [noise=true , address=104.36.82.46:4496 , piece_id=AB4YCF2ADLAAR6IA7NWUWEMG22RWISWYVTMF5P6F22LY74VXGU6A, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.132105816 (2h12m13.913039179s) [noise=true , address=77.90.31.16:4736 , piece_id=57P5D4LLH4L6DCMTDRSOX2BMWG6JQCTKC352SFWNC7RLMLLKLNAQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.132570113 (2h12m13.913503496s) [noise=true , address=79.127.201.215:4354 , piece_id=YEIJFT75CAWLYWRFAXYYD6O6A5VT6QFKMZS6FZRVCUFDO66MX5ZQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.133069101 (2h12m13.91400249s) [noise=true , address=104.36.82.52:4111 , piece_id=QRZ6PRVLMEXUECFWJRINPT6NQJ2ZCYPSNPG4TXSPPKGZLF6HPTXA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.133720330 (2h12m13.914653718s) [noise=true , address=104.36.82.42:4821 , piece_id=OIKULIQVGAQCWW4YGSTQLZ2HHPAEUSNGXCGZVBMXCLG4USNZ4QFQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.134209015 (2h12m13.915142406s) [noise=true , address=77.90.31.19:4871 , piece_id=S4OA45TJCK4L65QSKXHO6ZKQLDZZHHO7CINEQ7H52J5GGENG7NCQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.135012129 (2h12m13.915945519s) [noise=true , address=45.91.116.13:4157 , piece_id=DMB4A2Z7FJBM3MHJLNOKPGOWGKHRKI2ISEVHE45LI5JJQ3GLFVCA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.136811748 (2h12m13.917745121s) [noise=true , address=64.49.13.138:4482 , piece_id=S66BKIBJVE3O3XJJR6LVQUEWDDXYUMAIFG6C5NMDM5NWIVIAI65Q, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.137262345 (2h12m13.918195733s) [noise=true , address=64.49.13.137:4502 , piece_id=U66CNXLCTNMYSJPRHTMOCOEXPEGP6RJFBRKGDUSBHLVBPTBV47QQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.137965913 (2h12m13.91889931s) [noise=true , address=79.127.205.246:4282 , piece_id=P5OO4M32L2HTZTVKMDQ4F4MGXLXKPZI6PGL5CWWXS3ZYAOQOLRGA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.138576563 (2h12m13.919509954s) [noise=true , address=77.90.31.11:4574 , piece_id=ZYU43ARUUGC3HIVX7ADQP7CCLHNMLNU4BVCJPR6KHKW6DTYDPIFA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.139293589 (2h12m13.920226972s) [noise=true , address=79.127.219.38:4017 , piece_id=KNHHFET6O43QARQBBPIYLOJWAWJ7AXF6KHJPHWNGDTLJ5HEUJLZA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.139785639 (2h12m13.920719027s) [noise=true , address=77.90.31.21:4907 , piece_id=WN6EVB7WRX3KHDPYMP6D24Q3PWA3EKZCVHYSZ74RFDI3Y3Y7MYGA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.140380986 (2h12m13.921314373s) [noise=true , address=104.36.82.54:4175 , piece_id=BEUCXUYBV7BI6LZVUXDRFKYVAUSUQJ472LNQPE3C4K5VRBNWE5QA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.271977562 (2h12m14.05291094s) [upload=2025]: Next returned nil segment Jul 27 00:03:58.271994972 (2h12m14.052928349s) [upload=2025]: Waiting for error group managing segments... Jul 27 00:03:58.318044729 (2h12m14.098978112s) [noise=true , address=77.90.31.11:4574 , piece_id=ZYU43ARUUGC3HIVX7ADQP7CCLHNMLNU4BVCJPR6KHKW6DTYDPIFA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:03:58.339680307 (2h12m14.120613683s) [noise=true , address=77.90.31.21:4907 , piece_id=WN6EVB7WRX3KHDPYMP6D24Q3PWA3EKZCVHYSZ74RFDI3Y3Y7MYGA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:03:59.017719650 (2h12m14.798653016s) [noise=true , address=79.127.219.38:4017 , piece_id=KNHHFET6O43QARQBBPIYLOJWAWJ7AXF6KHJPHWNGDTLJ5HEUJLZA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:03:59.022755319 (2h12m14.803688681s) [noise=true , address=79.127.205.246:4282 , piece_id=P5OO4M32L2HTZTVKMDQ4F4MGXLXKPZI6PGL5CWWXS3ZYAOQOLRGA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:00.477236483 (2h12m16.258169845s) [noise=true , address=77.90.31.13:4636 , piece_id=EMBHHFZ3NKBGCO5E77DTO7QFR73CHRWLOYHZRAJJNRJSXYJBNIWA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:01.245636107 (2h12m17.02656947s) [noise=true , address=45.91.116.13:4157 , piece_id=DMB4A2Z7FJBM3MHJLNOKPGOWGKHRKI2ISEVHE45LI5JJQ3GLFVCA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:02.143012772 (2h12m17.923946152s) [noise=true , address=77.90.31.19:4871 , piece_id=S4OA45TJCK4L65QSKXHO6ZKQLDZZHHO7CINEQ7H52J5GGENG7NCQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:03.867893306 (2h12m19.648826683s) [noise=true , address=64.49.13.138:4482 , piece_id=S66BKIBJVE3O3XJJR6LVQUEWDDXYUMAIFG6C5NMDM5NWIVIAI65Q, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:05.225529650 (2h12m21.006463025s) [noise=true , address=64.49.13.137:4502 , piece_id=U66CNXLCTNMYSJPRHTMOCOEXPEGP6RJFBRKGDUSBHLVBPTBV47QQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:07.858858960 (2h12m23.639792332s) [noise=true , address=64.49.13.154:4927 , piece_id=3M4S26SJQTYS5WYS5B6KF6LGTYBXRHS2J6AHI26BOP4K5M22YXLQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:14.409102513 (2h12m30.190035875s) [noise=true , address=79.127.201.215:4354 , piece_id=YEIJFT75CAWLYWRFAXYYD6O6A5VT6QFKMZS6FZRVCUFDO66MX5ZQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:14.650696276 (2h12m30.431629641s) [noise=true , address=77.90.31.16:4736 , piece_id=57P5D4LLH4L6DCMTDRSOX2BMWG6JQCTKC352SFWNC7RLMLLKLNAQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:58.028785784 (2h13m13.80971916s) [noise=true , address=104.36.82.54:4170 , piece_id=BVFZN6P2XD6AEA34MZKT4MN7PPGVUHUEDT6RQFVPOIMHCL4NUGZQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: write tcp 159.223.143.39:35264->104.36.82.54:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.028868510 (2h13m13.809801875s) [noise=true , address=104.36.82.54:4175 , piece_id=BEUCXUYBV7BI6LZVUXDRFKYVAUSUQJ472LNQPE3C4K5VRBNWE5QA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: write tcp 159.223.143.39:34108->104.36.82.54:4175: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.029044515 (2h13m13.809977888s) [noise=true , address=104.36.82.46:4496 , piece_id=AB4YCF2ADLAAR6IA7NWUWEMG22RWISWYVTMF5P6F22LY74VXGU6A, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: write tcp 159.223.143.39:36988->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.029321639 (2h13m13.810255039s) [noise=true , address=104.36.82.42:4821 , piece_id=OIKULIQVGAQCWW4YGSTQLZ2HHPAEUSNGXCGZVBMXCLG4USNZ4QFQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:1Q2vaBr3JNY3jv8jVUTg9uzfWUtNjrd5c5Z9Yfi54D3JMRM1af, address:104.36.82.42:4821): protocol: write tcp 159.223.143.39:58022->104.36.82.42:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.029356741 (2h13m13.810290107s) [noise=true , address=104.36.82.52:4111 , piece_id=QRZ6PRVLMEXUECFWJRINPT6NQJ2ZCYPSNPG4TXSPPKGZLF6HPTXA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: write tcp 159.223.143.39:39766->104.36.82.52:4111: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.029396743 (2h13m13.810330112s) [noise=true , address=104.36.82.47:4549 , piece_id=6UDTTFYCYC5LBH5G4WM54UU6YRG2U4B65LONOHMOK3VIPKUTM7PQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:1HWoyMKQ6v2XWUfEmfFh3m8AiGivAcYkkZ9na8kqHFzH2hPDeK, address:104.36.82.47:4549): protocol: write tcp 159.223.143.39:55352->104.36.82.47:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.029437165 (2h13m13.810370556s) [seg_pos={0 0}, upload=2025]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:04:58.053486151 (2h13m13.834419518s) [upload=2025]: Done uploading segments... Jul 27 00:04:58.053533995 (2h13m13.834467361s) [upload=2025]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.027894959s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X.......X...X....XX..............X.X........X...XX..........] duration: 33.281110891s FAIL (2025-07-27 00:04:58 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.281131799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......XXX...XX.....X.XXX...............X..X......X.XX....X..] duration: 1.860637841s FAIL (2025-07-27 00:04:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.860942313s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX..] duration: 2.010817168s FAIL (2025-07-27 00:04:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:56.936460143 (2h11m12.717393528s) [upload=2012]: Starting upload Jul 27 00:02:56.936557479 (2h11m12.717490863s) [upload=2012]: Done starting upload Jul 27 00:02:56.936647830 (2h11m12.717581213s) [upload=2012]: Uploading segments... Jul 27 00:02:56.937017175 (2h11m12.71795055s) [upload=2012]: Got next segment. Inline: false Jul 27 00:02:56.937075780 (2h11m12.71800916s) [upload=2012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:56.937082791 (2h11m12.71801617s) [upload=2012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:56.970415476 (2h11m12.751348861s) [seg_pos={0 0}, upload=2012]: Begin upload segment... Jul 27 00:02:56.970488660 (2h11m12.751422026s) [seg_pos={0 0}, upload=2012]: Done begin upload segment. Jul 27 00:02:56.970499199 (2h11m12.751432579s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970507686 (2h11m12.751441073s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970999895 (2h11m12.75193328s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.971544420 (2h11m12.752477804s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974009791 (2h11m12.754943174s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974480217 (2h11m12.755413597s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974879774 (2h11m12.755813153s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975304158 (2h11m12.756237538s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975704939 (2h11m12.756638317s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976240365 (2h11m12.757173746s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976662963 (2h11m12.757596345s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977149854 (2h11m12.75808323s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977565879 (2h11m12.758499264s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977962142 (2h11m12.758895525s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978370394 (2h11m12.759303791s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978686570 (2h11m12.759619953s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:57.069515194 (2h11m12.850448562s) [upload=2012]: Next returned nil segment Jul 27 00:02:57.069531895 (2h11m12.850465271s) [upload=2012]: Waiting for error group managing segments... Jul 27 00:02:57.073615634 (2h11m12.854549001s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074286917 (2h11m12.855220285s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074641770 (2h11m12.855575146s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074773340 (2h11m12.855706716s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075293141 (2h11m12.856226509s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075311903 (2h11m12.856245271s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075673862 (2h11m12.856607227s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075747686 (2h11m12.856681054s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.139972458 (2h11m12.920905828s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.172759261 (2h11m12.953692623s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:58.458246223 (2h11m14.2391796s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:03:26.903729789 (2h11m42.684663172s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903813623 (2h11m42.684747001s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903844203 (2h11m42.684777573s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903891506 (2h11m42.684824887s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:35200->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903946088 (2h11m42.684879465s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903986605 (2h11m42.684919991s) [seg_pos={0 0}, upload=2012]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:03:26.932638927 (2h11m42.713572307s) [upload=2012]: Done uploading segments... Jul 27 00:03:26.932697222 (2h11m42.713630598s) [upload=2012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.010851346s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.X.......X...X....XX..............X.X........X...XX.........] duration: 29.661061768s FAIL (2025-07-27 00:04:25 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.661144258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXX] duration: 30.032421221s FAIL (2025-07-27 00:04:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.032448816s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [X.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.026825777s FAIL (2025-07-27 00:03:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 36.82.47:4520 , piece_id=XUBM7K4GBSNFCRI25KW5WZ3TB6JGRPJKDYD5WTHJAV7B4SALESIA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.017070783 (2h11m13.79800417s) [noise=true , address=79.127.219.38:4018 , piece_id=VLNZB5PH33GZTLQLVLQAQEPTTJSL5ERBN2BYKECAPRJHJUHI7TJA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.017538287 (2h11m13.798471665s) [noise=true , address=64.49.13.138:4483 , piece_id=Y44MF2GI4CXVIP3R7ZQYW2J5AB4ILO4AJNGJJSYLMPV3JY5VN5TA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.017549030 (2h11m13.798482418s) [noise=true , address=77.90.31.16:4748 , piece_id=ZDPCPBQHSBVWJFCZGEGUAANVZTGGCNMLWNRS3JDK4P3IE6H2FMYA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.017906052 (2h11m13.798839435s) [noise=true , address=79.127.205.235:4679 , piece_id=VZQFYJ5YH2UX6EQ5RODDKHE6QTYRM4O7AXNGH2WTAZ5CDW4G2P2Q, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.018004815 (2h11m13.798938211s) [noise=true , address=64.49.13.138:4482 , piece_id=YR3ZW2HGMKDKP32ZWO6KUK3DXF7VVIKDPXRW4XU6NEABNQK4ITCA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.018240986 (2h11m13.799174365s) [noise=true , address=104.36.82.52:4120 , piece_id=5AB6OGGGPSEDVO3PITQEAQYD7EJPQD2Q22QGT44UIYCSI2KJRCUA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.018593327 (2h11m13.799526714s) [noise=true , address=79.127.205.246:4269 , piece_id=NVAXJ4U4B6K3J46RYXANKMQLPB65CRIKI5M6RWSKTUS55EXJHFUQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.019074924 (2h11m13.800008307s) [noise=true , address=104.36.82.53:4148 , piece_id=NADXC75XGFLZPO5KVESTOAAGCDNTADGIK2ZURLKTNND7CNTGGLKA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.019518168 (2h11m13.800451553s) [noise=true , address=45.91.116.14:4206 , piece_id=PJLBYAJG3TH3YELB3VONBXEXNOB6FKXCIT4NG7VSEQ67KWQEK6LQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.020010537 (2h11m13.800943928s) [noise=true , address=104.36.82.54:4175 , piece_id=D3CRUPGTXPRDHVDUD3GOMHTT2DJC4WDI327CD7WNKMPDGJ5O5VIA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.020611094 (2h11m13.801544483s) [noise=true , address=79.127.201.215:4345 , piece_id=MTGMHK45SVSFEJX5EDKL3RLYRCFDLRHP2ZCMQZ5XF7LXAPBAONPQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.020998948 (2h11m13.801932338s) [noise=true , address=104.36.82.13:4211 , piece_id=2NO46AEDJK7Q3XPXIBWYZ2VENG345IRLGSBTVD7ZW4H7CTAZW7PQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.021489131 (2h11m13.802422521s) [noise=true , address=79.127.220.101:4048 , piece_id=3JTIL3DQRMYQMNGMRFIRNLWCZWXUQQMJQ2CWTLTSKO2ZNYVSBIQQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.021935419 (2h11m13.802868783s) [noise=true , address=104.36.82.20:4443 , piece_id=FMXBSLQRS5SG63CF5LHDTZMBTHZWEWBWHKELB7BU7NACFVVKZL4A, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.023066631 (2h11m13.804000023s) [noise=true , address=77.90.31.11:4579 , piece_id=R7ZSAQR7Z4VWDMTDJNVBIBDREEIHX5XEHMZ2I4OM2ENLNJD7GUBQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.023950660 (2h11m13.804884044s) [noise=true , address=104.36.82.46:4499 , piece_id=IGGQR5V36IQDZEXRIQW2NB4AH6XXMKAUUSTGONVUVMOGMG4LUSNA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.049779256 (2h11m13.830712646s) [noise=true , address=79.127.205.234:4618 , piece_id=EXLWSGG6FWHQRT53BDSECCVSQUG4BVSFYCDOWMR2CJTGAZK2ZDFA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.112187641 (2h11m13.89312102s) [upload=2013]: Next returned nil segment Jul 27 00:02:58.112207795 (2h11m13.893141182s) [upload=2013]: Waiting for error group managing segments... Jul 27 00:02:58.116592894 (2h11m13.897526281s) [noise=true , address=79.127.219.38:4018 , piece_id=VLNZB5PH33GZTLQLVLQAQEPTTJSL5ERBN2BYKECAPRJHJUHI7TJA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:02:58.154311610 (2h11m13.935244992s) [noise=true , address=77.90.31.11:4579 , piece_id=R7ZSAQR7Z4VWDMTDJNVBIBDREEIHX5XEHMZ2I4OM2ENLNJD7GUBQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:02:58.623605977 (2h11m14.404539339s) [noise=true , address=79.127.205.246:4269 , piece_id=NVAXJ4U4B6K3J46RYXANKMQLPB65CRIKI5M6RWSKTUS55EXJHFUQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:02:58.635272139 (2h11m14.416205507s) [noise=true , address=79.127.205.235:4679 , piece_id=VZQFYJ5YH2UX6EQ5RODDKHE6QTYRM4O7AXNGH2WTAZ5CDW4G2P2Q, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:02:58.680942641 (2h11m14.461876023s) [noise=true , address=79.127.205.234:4618 , piece_id=EXLWSGG6FWHQRT53BDSECCVSQUG4BVSFYCDOWMR2CJTGAZK2ZDFA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:02:59.109300394 (2h11m14.89023377s) [noise=true , address=79.127.220.101:4048 , piece_id=3JTIL3DQRMYQMNGMRFIRNLWCZWXUQQMJQ2CWTLTSKO2ZNYVSBIQQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:03:05.870854136 (2h11m21.651787504s) [noise=true , address=64.49.13.138:4483 , piece_id=Y44MF2GI4CXVIP3R7ZQYW2J5AB4ILO4AJNGJJSYLMPV3JY5VN5TA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:03:06.380469285 (2h11m22.161402667s) [noise=true , address=64.49.13.138:4482 , piece_id=YR3ZW2HGMKDKP32ZWO6KUK3DXF7VVIKDPXRW4XU6NEABNQK4ITCA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:03:08.016464682 (2h11m23.797398054s) [noise=true , address=45.91.116.14:4206 , piece_id=PJLBYAJG3TH3YELB3VONBXEXNOB6FKXCIT4NG7VSEQ67KWQEK6LQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:03:38.212561053 (2h11m53.993494435s) [noise=true , address=79.127.201.215:4345 , piece_id=MTGMHK45SVSFEJX5EDKL3RLYRCFDLRHP2ZCMQZ5XF7LXAPBAONPQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:03:57.938576320 (2h12m13.719509708s) [noise=true , address=77.90.31.16:4748 , piece_id=ZDPCPBQHSBVWJFCZGEGUAANVZTGGCNMLWNRS3JDK4P3IE6H2FMYA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: rpc: dial tcp 77.90.31.16:4748: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.938729802 (2h12m13.719663182s) [noise=true , address=104.36.82.52:4120 , piece_id=5AB6OGGGPSEDVO3PITQEAQYD7EJPQD2Q22QGT44UIYCSI2KJRCUA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 159.223.143.39:59044->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.938823589 (2h12m13.719756966s) [noise=true , address=104.36.82.54:4175 , piece_id=D3CRUPGTXPRDHVDUD3GOMHTT2DJC4WDI327CD7WNKMPDGJ5O5VIA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.938834539 (2h12m13.719767908s) [noise=true , address=104.36.82.46:4499 , piece_id=IGGQR5V36IQDZEXRIQW2NB4AH6XXMKAUUSTGONVUVMOGMG4LUSNA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: rpc: dial tcp 104.36.82.46:4499: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.938883165 (2h12m13.719816542s) [noise=true , address=104.36.82.47:4520 , piece_id=XUBM7K4GBSNFCRI25KW5WZ3TB6JGRPJKDYD5WTHJAV7B4SALESIA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: write tcp 159.223.143.39:40904->104.36.82.47:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.938964526 (2h12m13.719897908s) [noise=true , address=104.36.82.53:4148 , piece_id=NADXC75XGFLZPO5KVESTOAAGCDNTADGIK2ZURLKTNND7CNTGGLKA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: rpc: dial tcp 104.36.82.53:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.939008149 (2h12m13.719941527s) [noise=true , address=104.36.82.20:4443 , piece_id=FMXBSLQRS5SG63CF5LHDTZMBTHZWEWBWHKELB7BU7NACFVVKZL4A, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:1rTVUAPUj4PUy8qHuutnB2WYCVpJdiVwfutA6UK4TZoPzvQEUv, address:104.36.82.20:4443): protocol: write tcp 159.223.143.39:36038->104.36.82.20:4443: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.939074361 (2h12m13.720007738s) [noise=true , address=104.36.82.13:4211 , piece_id=2NO46AEDJK7Q3XPXIBWYZ2VENG345IRLGSBTVD7ZW4H7CTAZW7PQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:12dANoYTygpcpWfCcAJtsnVBFWSekLhDjheAXnsaYMChRYAm2qc, address:104.36.82.13:4211): protocol: write tcp 159.223.143.39:46864->104.36.82.13:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.939111306 (2h12m13.720044706s) [seg_pos={0 0}, upload=2013]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:03:57.961726382 (2h12m13.742659746s) [upload=2013]: Done uploading segments... Jul 27 00:03:57.961758648 (2h12m13.742692024s) [upload=2013]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.02684651s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)