Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 346.222654ms FAIL (2025-07-26 22:40:34 UTC) orbiter warning: uplink: too many requests Duration: 346.241481ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.920141279s FAIL (2025-07-26 22:40:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.920193145s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....XXX.XXX.X.XX..X..X......X.............X................] duration: 444.928328ms FAIL (2025-07-26 22:40:17 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 444.985143ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 555.803902ms FAIL (2025-07-26 22:40:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 555.82877ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XX] duration: 1m0.036278761s FAIL (2025-07-26 22:40:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== : Starting upload Jul 26 22:39:08.923788388 (47m24.704721774s) [upload=733]: Done starting upload Jul 26 22:39:08.924128497 (47m24.705061877s) [upload=733]: Uploading segments... Jul 26 22:39:08.924435891 (47m24.705369269s) [upload=733]: Got next segment. Inline: false Jul 26 22:39:08.924507771 (47m24.705441151s) [upload=733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:39:08.924514132 (47m24.705447515s) [upload=733]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:39:08.961235730 (47m24.742169115s) [seg_pos={0 0}, upload=733]: Begin upload segment... Jul 26 22:39:08.961321571 (47m24.742254956s) [seg_pos={0 0}, upload=733]: Done begin upload segment. Jul 26 22:39:08.961351688 (47m24.742285077s) [noise=true , address=79.127.201.215:4360 , piece_id=NGVCW4MLZSTK2TBEO7OLM7X5MZD6OFGBFJFQC2MCWBSNHRNVRQEQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.961370010 (47m24.742303392s) [noise=true , address=45.91.116.12:4229 , piece_id=WJ3LFHU36TPCZAVYLMWT3JFZF62C6LKBNOC4NZXUVQAQ6G2FESZQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.961860586 (47m24.742793968s) [noise=true , address=79.127.220.101:4048 , piece_id=X55VC4JDHS2FBPBCLNFDJ2OHD72WB65ZFHHGET54EUFHLOB77WDA, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.962269973 (47m24.743203356s) [noise=true , address=79.127.219.48:4108 , piece_id=ZD5SB7BTYATPL6GDJVMPXSLMQG3A2DUWIZPC7WIZSLRL6IOR2GGQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.962721691 (47m24.743655077s) [noise=true , address=77.90.31.12:4612 , piece_id=5BJ7IT4LNU7K2SSTR4VIBBY6KOAHQCXKYW44HA7UPC2KJTMHEY3A, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.963156595 (47m24.744089979s) [noise=true , address=77.90.31.16:4743 , piece_id=J7R3UT7IX5TE4NZIXVNC7UUO6RGELOYKTRG3DHZCDRPOAPPSL62A, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.963549149 (47m24.744482533s) [noise=true , address=156.146.43.225:4448 , piece_id=J7L3DC3T5LIAX4YXO4ZXMNNO5XUCDDMJKJEG5H42VJFOS7SYAQ4A, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.963954262 (47m24.744887645s) [noise=true , address=79.127.219.47:4069 , piece_id=F62NQJ5FL2UID4ZANGIYXVGRCMGIAB3DKMFVVRUQNSFWAW3CXRBQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.968141076 (47m24.749074452s) [noise=true , address=45.91.116.9:4268 , piece_id=I2LSWLBB5KBEX3T7G2VG7JY46LWGHFDAQALKMEYKCBPQ2QYTY2HQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.968650830 (47m24.74958422s) [noise=true , address=64.49.13.170:4721 , piece_id=CN5C77YJYS6EXXRZA2KLJ2DJBBU24IJMIIUBEVDXB224T5ILPMYA, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.969137867 (47m24.750071254s) [noise=true , address=45.91.116.10:4100 , piece_id=CVKC5XCMKVIEFG7MAWWU476NUSS46K7BW46DLCDGXCCXWW3DLTVQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.969641119 (47m24.750574506s) [noise=true , address=79.127.205.235:4682 , piece_id=QKAZRYX7JK3LPDKG7MEBZ3COLUFRYWC5O3BDSA332EOE3XALJ4IA, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.970186290 (47m24.751119677s) [noise=true , address=104.36.82.22:4509 , piece_id=QJWYVVJ2UYTMTECGJLDBUPBVSJQQBXKC55B5CAT3JFOCID36AJAQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.970591719 (47m24.751525106s) [noise=true , address=79.127.219.51:4210 , piece_id=XYEMCDISA33LQU33TIOEXC7TLCZFCPYY2VFNJ4GSJDQJD3PJKXZQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.970965884 (47m24.751899272s) [noise=true , address=79.127.205.245:4251 , piece_id=AI64TIXMPEG5AXKWS3IYOCW6ETQ5TNO2ZGEWXPQS2YUVRKNKHQ2Q, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.971409364 (47m24.752342751s) [noise=true , address=64.49.13.155:4923 , piece_id=QJTWA6ZXJASTJQYNLYGG3GTL6E2ZKAOYN6ZGZA7UUGNCT3TS2K7Q, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.971812382 (47m24.752745767s) [noise=true , address=79.127.219.38:4003 , piece_id=X2HHRAODDWWJU4BCSCZLR733BIETPPDHX52XXB3ANGYCQ7IJTXRA, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.974320352 (47m24.755253738s) [noise=true , address=79.127.205.234:4606 , piece_id=574OK4SH67GGE77YJJC2JWUWIAPAQK7GWZQ4WV63WAH7LGE23R6A, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:09.031814795 (47m24.81274818s) [noise=true , address=45.91.116.9:4268 , piece_id=I2LSWLBB5KBEX3T7G2VG7JY46LWGHFDAQALKMEYKCBPQ2QYTY2HQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, address:45.91.116.9:4268): protocol: not enough available disk space, have: 0, need: 6725632 Jul 26 22:39:09.032522272 (47m24.813455639s) [noise=true , address=79.127.219.48:4108 , piece_id=ZD5SB7BTYATPL6GDJVMPXSLMQG3A2DUWIZPC7WIZSLRL6IOR2GGQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:1JqrKuRY5vhNV26p42EdfHb5KePUEHnRAUQfna2RLSFz3XXfsG, address:79.127.219.48:4108): protocol: not enough available disk space, have: 0, need: 6725632 Jul 26 22:39:09.078623076 (47m24.859556461s) [upload=733]: Next returned nil segment Jul 26 22:39:09.078644987 (47m24.859578386s) [upload=733]: Waiting for error group managing segments... Jul 26 22:39:09.098122933 (47m24.879056296s) [noise=true , address=77.90.31.12:4612 , piece_id=5BJ7IT4LNU7K2SSTR4VIBBY6KOAHQCXKYW44HA7UPC2KJTMHEY3A, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:09.099379503 (47m24.88031287s) [noise=true , address=79.127.219.51:4210 , piece_id=XYEMCDISA33LQU33TIOEXC7TLCZFCPYY2VFNJ4GSJDQJD3PJKXZQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:39:09.121579166 (47m24.902512531s) [noise=true , address=77.90.31.16:4743 , piece_id=J7R3UT7IX5TE4NZIXVNC7UUO6RGELOYKTRG3DHZCDRPOAPPSL62A, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:09.125866269 (47m24.90679965s) [noise=true , address=79.127.219.38:4003 , piece_id=X2HHRAODDWWJU4BCSCZLR733BIETPPDHX52XXB3ANGYCQ7IJTXRA, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:09.599049558 (47m25.379982931s) [noise=true , address=79.127.205.245:4251 , piece_id=AI64TIXMPEG5AXKWS3IYOCW6ETQ5TNO2ZGEWXPQS2YUVRKNKHQ2Q, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:09.615181047 (47m25.396114416s) [noise=true , address=79.127.205.235:4682 , piece_id=QKAZRYX7JK3LPDKG7MEBZ3COLUFRYWC5O3BDSA332EOE3XALJ4IA, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:09.678010390 (47m25.458943779s) [noise=true , address=79.127.205.234:4606 , piece_id=574OK4SH67GGE77YJJC2JWUWIAPAQK7GWZQ4WV63WAH7LGE23R6A, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:10.408670184 (47m26.189603547s) [noise=true , address=79.127.220.101:4048 , piece_id=X55VC4JDHS2FBPBCLNFDJ2OHD72WB65ZFHHGET54EUFHLOB77WDA, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:10.500255954 (47m26.281189319s) [noise=true , address=79.127.219.47:4069 , piece_id=F62NQJ5FL2UID4ZANGIYXVGRCMGIAB3DKMFVVRUQNSFWAW3CXRBQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:10.657003674 (47m26.437937044s) [noise=true , address=156.146.43.225:4448 , piece_id=J7L3DC3T5LIAX4YXO4ZXMNNO5XUCDDMJKJEG5H42VJFOS7SYAQ4A, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:13.153669640 (47m28.934603018s) [noise=true , address=64.49.13.170:4721 , piece_id=CN5C77YJYS6EXXRZA2KLJ2DJBBU24IJMIIUBEVDXB224T5ILPMYA, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:39:13.224172652 (47m29.005106021s) [noise=true , address=64.49.13.155:4923 , piece_id=QJTWA6ZXJASTJQYNLYGG3GTL6E2ZKAOYN6ZGZA7UUGNCT3TS2K7Q, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:40.649544231 (47m56.430477616s) [noise=true , address=45.91.116.12:4229 , piece_id=WJ3LFHU36TPCZAVYLMWT3JFZF62C6LKBNOC4NZXUVQAQ6G2FESZQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: storage node overloaded, request limit: 2000 Jul 26 22:39:50.837446148 (48m6.618379515s) [noise=true , address=79.127.201.215:4360 , piece_id=NGVCW4MLZSTK2TBEO7OLM7X5MZD6OFGBFJFQC2MCWBSNHRNVRQEQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:40:08.882267994 (48m24.663201374s) [noise=true , address=104.36.82.22:4509 , piece_id=QJWYVVJ2UYTMTECGJLDBUPBVSJQQBXKC55B5CAT3JFOCID36AJAQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:104.36.82.22:4509): protocol: rpc: dial tcp 104.36.82.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:40:08.882762173 (48m24.663695535s) [noise=true , address=45.91.116.10:4100 , piece_id=CVKC5XCMKVIEFG7MAWWU476NUSS46K7BW46DLCDGXCCXWW3DLTVQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: write tcp 159.223.143.39:36046->45.91.116.10:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:40:08.882820611 (48m24.663754064s) [seg_pos={0 0}, upload=733]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:40:08.910675515 (48m24.69160888s) [upload=733]: Done uploading segments... Jul 26 22:40:08.910912462 (48m24.691845855s) [upload=733]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.036302273s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [............................X........X.X.X...XX.............] duration: 3.338275413s FAIL (2025-07-26 22:40:08 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.338306565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 204.774258ms FAIL (2025-07-26 22:39:58 UTC) orbiter warning: uplink: too many requests Duration: 204.803531ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX..XXX..X..XX..X..XX.X....X....X........X.................] duration: 415.12892ms FAIL (2025-07-26 22:39:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.159218ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 448.162479ms FAIL (2025-07-26 22:39:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 448.185926ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.............................X........X.X.X...XX............] duration: 10.377429309s FAIL (2025-07-26 22:39:38 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.377452609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 239.278635ms FAIL (2025-07-26 22:39:35 UTC) orbiter warning: uplink: too many requests Duration: 239.333907ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [......XXX.XXX.X.XX..X..X......X.............X...............] duration: 606.118021ms FAIL (2025-07-26 22:39:19 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 606.152692ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX..XXX..X..XX..X..XX.X....X....X........X................] duration: 471.616703ms FAIL (2025-07-26 22:39:18 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 471.654985ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.X] duration: 1m0.027899098s FAIL (2025-07-26 22:39:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 9636s) [upload=716]: Starting upload Jul 26 22:38:08.835390171 (46m24.616323563s) [upload=716]: Done starting upload Jul 26 22:38:08.835566870 (46m24.616500241s) [upload=716]: Uploading segments... Jul 26 22:38:08.835782913 (46m24.616716297s) [upload=716]: Got next segment. Inline: false Jul 26 22:38:08.835854662 (46m24.61678805s) [upload=716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:38:08.835862516 (46m24.616795901s) [upload=716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:38:08.873069108 (46m24.654002498s) [seg_pos={0 0}, upload=716]: Begin upload segment... Jul 26 22:38:08.873160866 (46m24.654094247s) [seg_pos={0 0}, upload=716]: Done begin upload segment. Jul 26 22:38:08.873209596 (46m24.654142977s) [noise=true , address=79.127.205.245:4248 , piece_id=N4BVIG4RJ2DM4IG2UY5Q5X4ZVQXHZ3MWCMEJ66GSE3ACZXB25HPQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.873808981 (46m24.654742364s) [noise=true , address=77.90.31.13:4634 , piece_id=NLH3ETOODFJ2H4AW4L2VKHH3Y7VCNIXWGJPAO37ACZPGZTGUM7WQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.874263438 (46m24.655196826s) [noise=true , address=156.146.43.225:4448 , piece_id=45LL457ULSE3WSAEENTOMCTUSZXP3G7XFBADGRXT6SEPARNBQK5Q, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.874606330 (46m24.655539697s) [noise=true , address=77.90.31.11:4585 , piece_id=BJBA64B4SMOPD4UH7NAVHOT3IA4ERXRREMIKZBLVGDJ5SZYUOVVQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.875045445 (46m24.655978835s) [noise=true , address=79.127.219.48:4090 , piece_id=DV7WFRTPG7EG55JGYTK4VA4YOKQRU3RBNBJFULZETF7NKVYDD2IQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.875494730 (46m24.656428097s) [noise=true , address=79.127.205.235:4688 , piece_id=5YAGJOEICJQCDDKS4GU6OCBJUTM5SN2HUA4DL6ADIEQEBHDWO4XA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.875823526 (46m24.656756914s) [noise=true , address=45.91.116.5:4310 , piece_id=QXWHPJJT6WFMORQXG3CDZ7TFNAFBFQDAXUKHU6W5WEAYYRWCRP4A, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.876383365 (46m24.657316753s) [noise=true , address=77.90.31.17:4779 , piece_id=TZZVMWKUK4VOOCYXHEKBVGDPB2TCSVTYDB4VD4PTFVWOB56MH4WA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.876740836 (46m24.657674222s) [noise=true , address=79.127.220.101:4048 , piece_id=6OV4FPE6SFDT6WU6JF3TAY5U4GMOBIK4MCJRMCPY7VA7CUZJ5B2Q, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.877379894 (46m24.658313282s) [noise=true , address=77.90.31.18:4839 , piece_id=CWJ4MSY6YK2AZ6XITE2WRON2T54XCYXIDFG5YEKOSBWRAZKUXV7Q, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.877749557 (46m24.658682947s) [noise=true , address=79.127.201.215:4360 , piece_id=5GZZQWBCWIMR455MQ257ETOZ2WEQ6BLYWLA6OHZIQ673EZS4OPCQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.878278845 (46m24.659212234s) [noise=true , address=79.127.219.38:4001 , piece_id=FH3V4IXPZCAQJ43AXVLL3XIEPJSQDPPPOI2ZWV72LL53U5UFFJNA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.878748085 (46m24.659681471s) [noise=true , address=77.90.31.15:4731 , piece_id=A6C73RYM67EGBOCLTAA764AZII636W4PKAZQ6DACRHJYK5RD5GUA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.879280496 (46m24.660213887s) [noise=true , address=79.127.205.246:4266 , piece_id=IM5NV6TLAMIUAWFLW6TVFWU45GRRMWNJAJMWZDQC6V7AT4NQOTLQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.879777351 (46m24.660710737s) [noise=true , address=77.90.31.21:4885 , piece_id=6RU2Y3S7UDHAJDHCHMFEYCMYZHOPU6JSVLXGYQE4WFYJXBP5HUEA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.880280029 (46m24.661213419s) [noise=true , address=193.149.164.143:4702 , piece_id=4HLD6BQVASDLVOKKHDS7GERYXLT43U2UXGJKISW73WF5ZRMIAQSQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.880664135 (46m24.661597516s) [noise=true , address=104.36.82.42:4826 , piece_id=ICX4E7GB2PYOJBILF3754FAL45TBSECQ6RTAH4PVDKM3THPKLZUA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.881101348 (46m24.662034742s) [noise=true , address=77.90.31.12:4608 , piece_id=5UZVWLWNFJGWBK7AAS4RNMLNU765GXNWFXDMCKFZHJDNEQKGGQOQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.945714663 (46m24.726648049s) [noise=true , address=79.127.219.48:4090 , piece_id=DV7WFRTPG7EG55JGYTK4VA4YOKQRU3RBNBJFULZETF7NKVYDD2IQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:08.945764441 (46m24.726697829s) [noise=true , address=77.90.31.13:4634 , piece_id=NLH3ETOODFJ2H4AW4L2VKHH3Y7VCNIXWGJPAO37ACZPGZTGUM7WQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, address:77.90.31.13:4634): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:09.004020033 (46m24.784953481s) [upload=716]: Next returned nil segment Jul 26 22:38:09.004042038 (46m24.784975425s) [upload=716]: Waiting for error group managing segments... Jul 26 22:38:09.022616432 (46m24.803549814s) [noise=true , address=156.146.43.225:4448 , piece_id=45LL457ULSE3WSAEENTOMCTUSZXP3G7XFBADGRXT6SEPARNBQK5Q, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:09.028632122 (46m24.809565484s) [noise=true , address=79.127.219.38:4001 , piece_id=FH3V4IXPZCAQJ43AXVLL3XIEPJSQDPPPOI2ZWV72LL53U5UFFJNA, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.043520465 (46m24.824453852s) [noise=true , address=79.127.220.101:4048 , piece_id=6OV4FPE6SFDT6WU6JF3TAY5U4GMOBIK4MCJRMCPY7VA7CUZJ5B2Q, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:09.052156291 (46m24.833089666s) [noise=true , address=77.90.31.21:4885 , piece_id=6RU2Y3S7UDHAJDHCHMFEYCMYZHOPU6JSVLXGYQE4WFYJXBP5HUEA, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.061288317 (46m24.842221681s) [noise=true , address=77.90.31.11:4585 , piece_id=BJBA64B4SMOPD4UH7NAVHOT3IA4ERXRREMIKZBLVGDJ5SZYUOVVQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.063141664 (46m24.844075051s) [noise=true , address=77.90.31.12:4608 , piece_id=5UZVWLWNFJGWBK7AAS4RNMLNU765GXNWFXDMCKFZHJDNEQKGGQOQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.093133770 (46m24.874067151s) [noise=true , address=77.90.31.17:4779 , piece_id=TZZVMWKUK4VOOCYXHEKBVGDPB2TCSVTYDB4VD4PTFVWOB56MH4WA, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.465880506 (46m25.246813874s) [noise=true , address=79.127.205.246:4266 , piece_id=IM5NV6TLAMIUAWFLW6TVFWU45GRRMWNJAJMWZDQC6V7AT4NQOTLQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.516371284 (46m25.297304661s) [noise=true , address=79.127.205.235:4688 , piece_id=5YAGJOEICJQCDDKS4GU6OCBJUTM5SN2HUA4DL6ADIEQEBHDWO4XA, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.517320708 (46m25.298254089s) [noise=true , address=79.127.205.245:4248 , piece_id=N4BVIG4RJ2DM4IG2UY5Q5X4ZVQXHZ3MWCMEJ66GSE3ACZXB25HPQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.641480962 (46m25.422414347s) [noise=true , address=193.149.164.143:4702 , piece_id=4HLD6BQVASDLVOKKHDS7GERYXLT43U2UXGJKISW73WF5ZRMIAQSQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:22.794780350 (46m38.575713727s) [noise=true , address=77.90.31.15:4731 , piece_id=A6C73RYM67EGBOCLTAA764AZII636W4PKAZQ6DACRHJYK5RD5GUA, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:39:05.978548585 (47m21.759481954s) [noise=true , address=79.127.201.215:4360 , piece_id=5GZZQWBCWIMR455MQ257ETOZ2WEQ6BLYWLA6OHZIQ673EZS4OPCQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:39:06.046009209 (47m21.826942594s) [noise=true , address=77.90.31.18:4839 , piece_id=CWJ4MSY6YK2AZ6XITE2WRON2T54XCYXIDFG5YEKOSBWRAZKUXV7Q, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx, address:77.90.31.18:4839): protocol: storage node overloaded, request limit: 2000 Jul 26 22:39:08.795713604 (47m24.576646988s) [noise=true , address=45.91.116.5:4310 , piece_id=QXWHPJJT6WFMORQXG3CDZ7TFNAFBFQDAXUKHU6W5WEAYYRWCRP4A, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: rpc: dial tcp 45.91.116.5:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:39:08.795824905 (47m24.576758299s) [noise=true , address=104.36.82.42:4826 , piece_id=ICX4E7GB2PYOJBILF3754FAL45TBSECQ6RTAH4PVDKM3THPKLZUA, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:13KQwy5RDTnWanKbCdnsdqRt88m58bEqcLmT3SpLxYW2uANXGv, address:104.36.82.42:4826): protocol: write tcp 159.223.143.39:59824->104.36.82.42:4826: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:39:08.795868246 (47m24.576801635s) [seg_pos={0 0}, upload=716]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:39:08.820571335 (47m24.601504707s) [upload=716]: Done uploading segments... Jul 26 22:39:08.820612436 (47m24.601545814s) [upload=716]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.027940729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......XXX.XXX.X.XX..X..X......X.............X..............] duration: 473.050039ms FAIL (2025-07-26 22:38:59 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 473.082686ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 474.399669ms FAIL (2025-07-26 22:38:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.685858ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..............................X........X.X.X...XX...........] duration: 3.478931534s FAIL (2025-07-26 22:38:55 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478960013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 443.537956ms FAIL (2025-07-26 22:38:34 UTC) orbiter warning: uplink: too many requests Duration: 443.562905ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...XX..XXX..X..XX..X..XX.X....X....X........X...............] duration: 413.935643ms FAIL (2025-07-26 22:38:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 413.970188ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [........XXX.XXX.X.XX..X..X......X.............X.............] duration: 636.568534ms FAIL (2025-07-26 22:38:19 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 636.69584ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....XX..XXX..X..XX..X..XX.X....X....X........X..............] duration: 672.576637ms FAIL (2025-07-26 22:38:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 672.617658ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.] duration: 15.484074278s FAIL (2025-07-26 22:38:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 22:36:43.901527855 (44m59.682461222s) [upload=691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:36:43.901542041 (44m59.682475408s) [upload=691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:36:43.942644971 (44m59.723578354s) [seg_pos={0 0}, upload=691]: Begin upload segment... Jul 26 22:36:43.942715586 (44m59.723648966s) [seg_pos={0 0}, upload=691]: Done begin upload segment. Jul 26 22:36:43.942752085 (44m59.723685471s) [noise=true , address=79.127.205.246:4287 , piece_id=AHNHIJ6LYZ2N7UZXOCLJP37IXFC7GFFWIZLJHVELORER5AUI27LA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.943373384 (44m59.724306777s) [noise=true , address=45.91.116.10:4100 , piece_id=XYDVDHRMMD5SYULA6TL2P2EEWLUQDGRVUQJ64OFIBG5QFYIIJRVA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.943838441 (44m59.72477183s) [noise=true , address=104.36.82.45:4936 , piece_id=LSFD24OKUPW3MA5XIEYAUUVM2IPISVT3R7IPSVKRCIRUSMT4JPJA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.944268010 (44m59.725201382s) [noise=true , address=193.149.164.143:4702 , piece_id=N3BGCB6E3VHAKMMNVKH4TDY3VE3LESIYFCLWPYIZIRWWCNVSYNVQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.944678500 (44m59.72561189s) [noise=true , address=79.127.219.47:4061 , piece_id=HVS44Q3QXKXNVJ3FRPVOKPV2ITJW5HIUUG7CT627LML6HJPNBMIQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945065556 (44m59.725998962s) [noise=true , address=104.36.82.34:4565 , piece_id=E5TVWQSLOW36QYLZJPX3B35AFSUWIKVQ7BKVPXTLW3TQ7CKZIWIQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945462411 (44m59.726395801s) [noise=true , address=79.127.219.51:4210 , piece_id=HRYS7V5THQGB2ICXY3Z3FRWBTZIEYLAAOYBVONHOKNIS7MGZCZXA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945856178 (44m59.726789567s) [noise=true , address=77.90.31.11:4558 , piece_id=X4EWCLG4OZDNRWSTL5JGBOWIIAYY7FHCZPXY7IILV4ZFOVQRXSVQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.946244188 (44m59.727177623s) [noise=true , address=79.127.205.245:4221 , piece_id=HASU6Q44AMAEJ3FMRYFMVMRNRWVWHXQ6S4FREG4BQ2ZADBS44CBQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.946660379 (44m59.727593771s) [noise=true , address=156.146.43.225:4448 , piece_id=R3VG6RUKSNVZUQTLDSN34CT3OCDE2YL22AN6EODCVK44P32LZKQQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.947097799 (44m59.728031189s) [noise=true , address=79.127.205.233:4656 , piece_id=HP4XRH5CWO7ZM7VREM7PGQB2VB7RG2ZPWSN33YDASOFMBXOZLTGQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.950911616 (44m59.731845003s) [noise=true , address=79.127.201.215:4341 , piece_id=6PUJV5VF37C4U7I2Z42HMUCJHHBI5KRD6RDMJMHX6KVVQFLSTFHQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.951551416 (44m59.732484802s) [noise=true , address=77.90.31.15:4732 , piece_id=KFS73H6RQZVH7RQDXRPUOS54FGQNW4PUUQEPQCL44G4H322Y2LGQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.952021135 (44m59.73295452s) [noise=true , address=79.127.219.38:4017 , piece_id=S5AFCQCIRJCFX734MBO6PB7C4O6A5JRILR5XQFPVG4SIM4AQJ5OQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.952615232 (44m59.733548621s) [noise=true , address=79.127.205.234:4618 , piece_id=KA4DCNR2M7HIZBRDZCO5JXJVR4HEJMJ5PWUOGHVDPN224Z6DSSKA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.953194884 (44m59.734128265s) [noise=true , address=77.90.31.18:4831 , piece_id=QS3CARJPIZIZCJQTHON6DHEQEKIFCYDCKCGIVFTE3AZDOAVAZISQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.953919127 (44m59.734852514s) [noise=true , address=64.49.13.170:4719 , piece_id=5XGJZB2F4RRZ57PFHNLPGXEAUOJRN3YD7HHKIHU65G3E6NVLGTKQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.954395925 (44m59.735329313s) [noise=true , address=79.127.205.235:4688 , piece_id=4GFOOOBMY6HN3Y7HQV72Y53X4RN34KHF6Z56CGJUX3CJIWRZ2NLQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:44.048957514 (44m59.829890894s) [upload=691]: Next returned nil segment Jul 26 22:36:44.048977726 (44m59.829911114s) [upload=691]: Waiting for error group managing segments... Jul 26 22:36:44.053653368 (44m59.834586742s) [noise=true , address=79.127.219.38:4017 , piece_id=S5AFCQCIRJCFX734MBO6PB7C4O6A5JRILR5XQFPVG4SIM4AQJ5OQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.076905665 (44m59.857839032s) [noise=true , address=77.90.31.11:4558 , piece_id=X4EWCLG4OZDNRWSTL5JGBOWIIAYY7FHCZPXY7IILV4ZFOVQRXSVQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.112521058 (44m59.893454439s) [noise=true , address=156.146.43.225:4448 , piece_id=R3VG6RUKSNVZUQTLDSN34CT3OCDE2YL22AN6EODCVK44P32LZKQQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:44.550215284 (45m0.331148671s) [noise=true , address=79.127.205.234:4618 , piece_id=KA4DCNR2M7HIZBRDZCO5JXJVR4HEJMJ5PWUOGHVDPN224Z6DSSKA, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.566685558 (45m0.347618929s) [noise=true , address=79.127.205.245:4221 , piece_id=HASU6Q44AMAEJ3FMRYFMVMRNRWVWHXQ6S4FREG4BQ2ZADBS44CBQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.583061209 (45m0.363994591s) [noise=true , address=79.127.205.233:4656 , piece_id=HP4XRH5CWO7ZM7VREM7PGQB2VB7RG2ZPWSN33YDASOFMBXOZLTGQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.612532419 (45m0.393465802s) [noise=true , address=79.127.219.51:4210 , piece_id=HRYS7V5THQGB2ICXY3Z3FRWBTZIEYLAAOYBVONHOKNIS7MGZCZXA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:44.647749987 (45m0.428683349s) [noise=true , address=79.127.205.246:4287 , piece_id=AHNHIJ6LYZ2N7UZXOCLJP37IXFC7GFFWIZLJHVELORER5AUI27LA, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.764033389 (45m0.544966768s) [noise=true , address=79.127.205.235:4688 , piece_id=4GFOOOBMY6HN3Y7HQV72Y53X4RN34KHF6Z56CGJUX3CJIWRZ2NLQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:45.359283634 (45m1.140217014s) [noise=true , address=45.91.116.10:4100 , piece_id=XYDVDHRMMD5SYULA6TL2P2EEWLUQDGRVUQJ64OFIBG5QFYIIJRVA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:46.520984499 (45m2.301917869s) [noise=true , address=79.127.201.215:4341 , piece_id=6PUJV5VF37C4U7I2Z42HMUCJHHBI5KRD6RDMJMHX6KVVQFLSTFHQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:46.832078832 (45m2.613012203s) [noise=true , address=79.127.219.47:4061 , piece_id=HVS44Q3QXKXNVJ3FRPVOKPV2ITJW5HIUUG7CT627LML6HJPNBMIQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:47.033893712 (45m2.814827102s) [noise=true , address=193.149.164.143:4702 , piece_id=N3BGCB6E3VHAKMMNVKH4TDY3VE3LESIYFCLWPYIZIRWWCNVSYNVQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:48.145235648 (45m3.926169033s) [noise=true , address=64.49.13.170:4719 , piece_id=5XGJZB2F4RRZ57PFHNLPGXEAUOJRN3YD7HHKIHU65G3E6NVLGTKQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:37:43.869125014 (45m59.65005838s) [noise=true , address=104.36.82.45:4936 , piece_id=LSFD24OKUPW3MA5XIEYAUUVM2IPISVT3R7IPSVKRCIRUSMT4JPJA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8, address:104.36.82.45:4936): protocol: write tcp 159.223.143.39:57056->104.36.82.45:4936: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869315595 (45m59.65024897s) [noise=true , address=77.90.31.15:4732 , piece_id=KFS73H6RQZVH7RQDXRPUOS54FGQNW4PUUQEPQCL44G4H322Y2LGQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: write tcp 159.223.143.39:40916->77.90.31.15:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869502930 (45m59.650436305s) [noise=true , address=77.90.31.18:4831 , piece_id=QS3CARJPIZIZCJQTHON6DHEQEKIFCYDCKCGIVFTE3AZDOAVAZISQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 159.223.143.39:58740->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869645094 (45m59.650578461s) [noise=true , address=104.36.82.34:4565 , piece_id=E5TVWQSLOW36QYLZJPX3B35AFSUWIKVQ7BKVPXTLW3TQ7CKZIWIQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1JRxcRCxwuNJrRRcKKxEggFpmRk2fo9kdVbmhPn3eLLeiQop9R, address:104.36.82.34:4565): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:37:43.869697588 (45m59.650630973s) [seg_pos={0 0}, upload=691]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:37:43.904972836 (45m59.685906213s) [upload=691]: Done uploading segments... Jul 26 22:37:43.905001091 (45m59.685934467s) [upload=691]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 15.484107057s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 629.949881ms FAIL (2025-07-26 22:37:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 630.062255ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...............................X........X.X.X...XX..........] duration: 9.359107455s FAIL (2025-07-26 22:37:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.359864808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [X........XXX.XXX.X.XX..X..X......X.............X............] duration: 644.579124ms FAIL (2025-07-26 22:37:46 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 644.595811ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX] duration: 1m0.037908954s FAIL (2025-07-26 22:37:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 22:36:43.901527855 (44m59.682461222s) [upload=691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:36:43.901542041 (44m59.682475408s) [upload=691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:36:43.942644971 (44m59.723578354s) [seg_pos={0 0}, upload=691]: Begin upload segment... Jul 26 22:36:43.942715586 (44m59.723648966s) [seg_pos={0 0}, upload=691]: Done begin upload segment. Jul 26 22:36:43.942752085 (44m59.723685471s) [noise=true , address=79.127.205.246:4287 , piece_id=AHNHIJ6LYZ2N7UZXOCLJP37IXFC7GFFWIZLJHVELORER5AUI27LA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.943373384 (44m59.724306777s) [noise=true , address=45.91.116.10:4100 , piece_id=XYDVDHRMMD5SYULA6TL2P2EEWLUQDGRVUQJ64OFIBG5QFYIIJRVA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.943838441 (44m59.72477183s) [noise=true , address=104.36.82.45:4936 , piece_id=LSFD24OKUPW3MA5XIEYAUUVM2IPISVT3R7IPSVKRCIRUSMT4JPJA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.944268010 (44m59.725201382s) [noise=true , address=193.149.164.143:4702 , piece_id=N3BGCB6E3VHAKMMNVKH4TDY3VE3LESIYFCLWPYIZIRWWCNVSYNVQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.944678500 (44m59.72561189s) [noise=true , address=79.127.219.47:4061 , piece_id=HVS44Q3QXKXNVJ3FRPVOKPV2ITJW5HIUUG7CT627LML6HJPNBMIQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945065556 (44m59.725998962s) [noise=true , address=104.36.82.34:4565 , piece_id=E5TVWQSLOW36QYLZJPX3B35AFSUWIKVQ7BKVPXTLW3TQ7CKZIWIQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945462411 (44m59.726395801s) [noise=true , address=79.127.219.51:4210 , piece_id=HRYS7V5THQGB2ICXY3Z3FRWBTZIEYLAAOYBVONHOKNIS7MGZCZXA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945856178 (44m59.726789567s) [noise=true , address=77.90.31.11:4558 , piece_id=X4EWCLG4OZDNRWSTL5JGBOWIIAYY7FHCZPXY7IILV4ZFOVQRXSVQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.946244188 (44m59.727177623s) [noise=true , address=79.127.205.245:4221 , piece_id=HASU6Q44AMAEJ3FMRYFMVMRNRWVWHXQ6S4FREG4BQ2ZADBS44CBQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.946660379 (44m59.727593771s) [noise=true , address=156.146.43.225:4448 , piece_id=R3VG6RUKSNVZUQTLDSN34CT3OCDE2YL22AN6EODCVK44P32LZKQQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.947097799 (44m59.728031189s) [noise=true , address=79.127.205.233:4656 , piece_id=HP4XRH5CWO7ZM7VREM7PGQB2VB7RG2ZPWSN33YDASOFMBXOZLTGQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.950911616 (44m59.731845003s) [noise=true , address=79.127.201.215:4341 , piece_id=6PUJV5VF37C4U7I2Z42HMUCJHHBI5KRD6RDMJMHX6KVVQFLSTFHQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.951551416 (44m59.732484802s) [noise=true , address=77.90.31.15:4732 , piece_id=KFS73H6RQZVH7RQDXRPUOS54FGQNW4PUUQEPQCL44G4H322Y2LGQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.952021135 (44m59.73295452s) [noise=true , address=79.127.219.38:4017 , piece_id=S5AFCQCIRJCFX734MBO6PB7C4O6A5JRILR5XQFPVG4SIM4AQJ5OQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.952615232 (44m59.733548621s) [noise=true , address=79.127.205.234:4618 , piece_id=KA4DCNR2M7HIZBRDZCO5JXJVR4HEJMJ5PWUOGHVDPN224Z6DSSKA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.953194884 (44m59.734128265s) [noise=true , address=77.90.31.18:4831 , piece_id=QS3CARJPIZIZCJQTHON6DHEQEKIFCYDCKCGIVFTE3AZDOAVAZISQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.953919127 (44m59.734852514s) [noise=true , address=64.49.13.170:4719 , piece_id=5XGJZB2F4RRZ57PFHNLPGXEAUOJRN3YD7HHKIHU65G3E6NVLGTKQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.954395925 (44m59.735329313s) [noise=true , address=79.127.205.235:4688 , piece_id=4GFOOOBMY6HN3Y7HQV72Y53X4RN34KHF6Z56CGJUX3CJIWRZ2NLQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:44.048957514 (44m59.829890894s) [upload=691]: Next returned nil segment Jul 26 22:36:44.048977726 (44m59.829911114s) [upload=691]: Waiting for error group managing segments... Jul 26 22:36:44.053653368 (44m59.834586742s) [noise=true , address=79.127.219.38:4017 , piece_id=S5AFCQCIRJCFX734MBO6PB7C4O6A5JRILR5XQFPVG4SIM4AQJ5OQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.076905665 (44m59.857839032s) [noise=true , address=77.90.31.11:4558 , piece_id=X4EWCLG4OZDNRWSTL5JGBOWIIAYY7FHCZPXY7IILV4ZFOVQRXSVQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.112521058 (44m59.893454439s) [noise=true , address=156.146.43.225:4448 , piece_id=R3VG6RUKSNVZUQTLDSN34CT3OCDE2YL22AN6EODCVK44P32LZKQQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:44.550215284 (45m0.331148671s) [noise=true , address=79.127.205.234:4618 , piece_id=KA4DCNR2M7HIZBRDZCO5JXJVR4HEJMJ5PWUOGHVDPN224Z6DSSKA, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.566685558 (45m0.347618929s) [noise=true , address=79.127.205.245:4221 , piece_id=HASU6Q44AMAEJ3FMRYFMVMRNRWVWHXQ6S4FREG4BQ2ZADBS44CBQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.583061209 (45m0.363994591s) [noise=true , address=79.127.205.233:4656 , piece_id=HP4XRH5CWO7ZM7VREM7PGQB2VB7RG2ZPWSN33YDASOFMBXOZLTGQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.612532419 (45m0.393465802s) [noise=true , address=79.127.219.51:4210 , piece_id=HRYS7V5THQGB2ICXY3Z3FRWBTZIEYLAAOYBVONHOKNIS7MGZCZXA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:44.647749987 (45m0.428683349s) [noise=true , address=79.127.205.246:4287 , piece_id=AHNHIJ6LYZ2N7UZXOCLJP37IXFC7GFFWIZLJHVELORER5AUI27LA, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.764033389 (45m0.544966768s) [noise=true , address=79.127.205.235:4688 , piece_id=4GFOOOBMY6HN3Y7HQV72Y53X4RN34KHF6Z56CGJUX3CJIWRZ2NLQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:45.359283634 (45m1.140217014s) [noise=true , address=45.91.116.10:4100 , piece_id=XYDVDHRMMD5SYULA6TL2P2EEWLUQDGRVUQJ64OFIBG5QFYIIJRVA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:46.520984499 (45m2.301917869s) [noise=true , address=79.127.201.215:4341 , piece_id=6PUJV5VF37C4U7I2Z42HMUCJHHBI5KRD6RDMJMHX6KVVQFLSTFHQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:46.832078832 (45m2.613012203s) [noise=true , address=79.127.219.47:4061 , piece_id=HVS44Q3QXKXNVJ3FRPVOKPV2ITJW5HIUUG7CT627LML6HJPNBMIQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:47.033893712 (45m2.814827102s) [noise=true , address=193.149.164.143:4702 , piece_id=N3BGCB6E3VHAKMMNVKH4TDY3VE3LESIYFCLWPYIZIRWWCNVSYNVQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:48.145235648 (45m3.926169033s) [noise=true , address=64.49.13.170:4719 , piece_id=5XGJZB2F4RRZ57PFHNLPGXEAUOJRN3YD7HHKIHU65G3E6NVLGTKQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:37:43.869125014 (45m59.65005838s) [noise=true , address=104.36.82.45:4936 , piece_id=LSFD24OKUPW3MA5XIEYAUUVM2IPISVT3R7IPSVKRCIRUSMT4JPJA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8, address:104.36.82.45:4936): protocol: write tcp 159.223.143.39:57056->104.36.82.45:4936: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869315595 (45m59.65024897s) [noise=true , address=77.90.31.15:4732 , piece_id=KFS73H6RQZVH7RQDXRPUOS54FGQNW4PUUQEPQCL44G4H322Y2LGQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: write tcp 159.223.143.39:40916->77.90.31.15:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869502930 (45m59.650436305s) [noise=true , address=77.90.31.18:4831 , piece_id=QS3CARJPIZIZCJQTHON6DHEQEKIFCYDCKCGIVFTE3AZDOAVAZISQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 159.223.143.39:58740->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869645094 (45m59.650578461s) [noise=true , address=104.36.82.34:4565 , piece_id=E5TVWQSLOW36QYLZJPX3B35AFSUWIKVQ7BKVPXTLW3TQ7CKZIWIQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1JRxcRCxwuNJrRRcKKxEggFpmRk2fo9kdVbmhPn3eLLeiQop9R, address:104.36.82.34:4565): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:37:43.869697588 (45m59.650630973s) [seg_pos={0 0}, upload=691]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:37:43.904972836 (45m59.685906213s) [upload=691]: Done uploading segments... Jul 26 22:37:43.905001091 (45m59.685934467s) [upload=691]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.037938429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 242.71597ms FAIL (2025-07-26 22:37:24 UTC) orbiter warning: uplink: too many requests Duration: 242.76039ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [................................X........X.X.X...XX.........] duration: 23.497625938s FAIL (2025-07-26 22:37:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.497646183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....XX..XXX..X..XX..X..XX.X....X....X........X.............] duration: 498.386449ms FAIL (2025-07-26 22:37:05 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 498.419584ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 499.170886ms FAIL (2025-07-26 22:36:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 499.197325ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......XX..XXX..X..XX..X..XX.X....X....X........X............] duration: 748.695551ms FAIL (2025-07-26 22:36:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 748.726315ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X........XXX.XXX.X.XX..X..X......X.............X...........] duration: 517.439325ms FAIL (2025-07-26 22:36:45 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 517.478145ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.................................X........X.X.X...XX........] duration: 43.772930232s FAIL (2025-07-26 22:36:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.77296209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 271.915213ms FAIL (2025-07-26 22:36:43 UTC) orbiter warning: uplink: too many requests Duration: 271.945406ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 40 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XX] duration: 1m0.034621651s FAIL (2025-07-26 22:36:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 6803691s) [upload=675]: Uploading segments... Jul 26 22:35:30.687181937 (43m46.46811533s) [upload=675]: Got next segment. Inline: false Jul 26 22:35:30.687233799 (43m46.468167169s) [upload=675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:35:30.687239765 (43m46.46817315s) [upload=675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:35:30.728984867 (43m46.509918258s) [seg_pos={0 0}, upload=675]: Begin upload segment... Jul 26 22:35:30.729061688 (43m46.509995069s) [seg_pos={0 0}, upload=675]: Done begin upload segment. Jul 26 22:35:30.729097186 (43m46.510030574s) [noise=true , address=79.127.219.47:4048 , piece_id=QRZOSWTJLWGVEAU22CQIVRAXL7Q36XXAKU5QUKEUC342HZCNSMVQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.729652133 (43m46.510585522s) [noise=true , address=79.127.220.101:4048 , piece_id=IUPOOQESSZ23AXCOSV7QBJ7575JUZXEORLSTXMSWHXBTBD7ZZG2Q, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.730062086 (43m46.510995472s) [noise=true , address=79.127.219.38:4001 , piece_id=IHZPK256ADZE6FBYZ2SPXTZLDLVHF2QQYXH3UJ3XNABDDC4Q6NMA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.730451131 (43m46.511384519s) [noise=true , address=77.90.31.17:4787 , piece_id=LXM5IHAXGXEVSCEMBRZOLBLH6QW4KFNVNZ453PBO2BGTQ2UEKDIA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.730838247 (43m46.511771633s) [noise=true , address=79.127.219.51:4212 , piece_id=KR2627EFSL37ZN7DY2NK27COAEJN2SZPVE4VM733RJDWG4YWQHMQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.731256076 (43m46.512189465s) [noise=true , address=77.90.31.16:4743 , piece_id=BHEZR3DMDDJ4LMFKTYSWFY2PJOGY4QBG3N2R43UAPYEANZSD7XLQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.731686837 (43m46.512620226s) [noise=true , address=77.90.31.18:4831 , piece_id=AZM43A6FZCVL57DNBZNYTV6H52XZNRHGLUULM2SXOVF3KEQ4USWQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.732423866 (43m46.513357255s) [noise=true , address=77.90.31.19:4867 , piece_id=LNNOECSGUO4GYMOQPFMLX3SXDHKQCLAXKPEQNDWVDWVKZ5S4VOQA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.732872501 (43m46.513805888s) [noise=true , address=79.127.219.48:4090 , piece_id=Q64O75TAH4CS2OGAOMWVR76RHCQNF2AVVLOB7OHDCBJ2FMS2R4UQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.733254781 (43m46.51418817s) [noise=true , address=79.127.205.246:4285 , piece_id=CHRRUNCJDF4HSPPBLJLKSZDR4XOJPQCWAPVWBZWH2E7DZXH7DXWQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.733664752 (43m46.51459813s) [noise=true , address=79.127.205.234:4601 , piece_id=AISC6QJ44WOHECCE7OVBV7X6WRRVX4EXESCHTJR3IEBGMUC7AL3Q, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.734139169 (43m46.515072536s) [noise=true , address=64.49.13.170:4721 , piece_id=RXL6GSXCBSTSPFRAEY3O73XGXXVDFGCLJEPDKPLJ3RJU3JGTES2A, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.734553484 (43m46.515486876s) [noise=true , address=45.91.116.9:4269 , piece_id=VNR7SCF6KMHXULRFZKSZV7A7VHCEX2LN4YBFQHCOWQ7GO5X3IW4Q, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.734963101 (43m46.51589649s) [noise=true , address=104.36.82.11:4138 , piece_id=PVYONMLKHQX6HGCM45ZTN2TAJMXOAM6PNURSBPRHNRF7FTUQI4TA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.735421554 (43m46.516354943s) [noise=true , address=79.127.201.215:4359 , piece_id=ZNSGRGVJWNKHZPQG5BXNY7M6NBOX7J34BOFLJVLKFNPPSEOUKY4Q, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.735828597 (43m46.516761985s) [noise=true , address=77.90.31.15:4723 , piece_id=XT32Y4DX554I3KVOCKS6NTIP3ASCQSAUPLV5NXG66WEGH7TENTNA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.736270428 (43m46.517203815s) [noise=true , address=64.49.13.138:4482 , piece_id=LRNWJIBYUO4LEXHXBOB6EDY7QVTLQHCBWF4Q4CL5PXTKSFT7WDIA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.736674151 (43m46.517607543s) [noise=true , address=156.146.43.225:4448 , piece_id=YEHPZKKHQLJ57VOK6RENN4PFNJ6QYHN26Z3T47C4D6EM67Y4OOKA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.840412650 (43m46.621346033s) [upload=675]: Next returned nil segment Jul 26 22:35:30.840434059 (43m46.621367444s) [upload=675]: Waiting for error group managing segments... Jul 26 22:35:30.887100731 (43m46.668034109s) [noise=true , address=77.90.31.17:4787 , piece_id=LXM5IHAXGXEVSCEMBRZOLBLH6QW4KFNVNZ453PBO2BGTQ2UEKDIA, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:30.974047219 (43m46.754980597s) [noise=true , address=79.127.219.38:4001 , piece_id=IHZPK256ADZE6FBYZ2SPXTZLDLVHF2QQYXH3UJ3XNABDDC4Q6NMA, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:31.012374471 (43m46.793307855s) [noise=true , address=79.127.219.51:4212 , piece_id=KR2627EFSL37ZN7DY2NK27COAEJN2SZPVE4VM733RJDWG4YWQHMQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:35:31.349945905 (43m47.13087928s) [noise=true , address=79.127.205.234:4601 , piece_id=AISC6QJ44WOHECCE7OVBV7X6WRRVX4EXESCHTJR3IEBGMUC7AL3Q, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:32.028901166 (43m47.809834536s) [noise=true , address=79.127.219.48:4090 , piece_id=Q64O75TAH4CS2OGAOMWVR76RHCQNF2AVVLOB7OHDCBJ2FMS2R4UQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:32.175720135 (43m47.956653511s) [noise=true , address=45.91.116.9:4269 , piece_id=VNR7SCF6KMHXULRFZKSZV7A7VHCEX2LN4YBFQHCOWQ7GO5X3IW4Q, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:35:32.314503227 (43m48.095436606s) [noise=true , address=79.127.220.101:4048 , piece_id=IUPOOQESSZ23AXCOSV7QBJ7575JUZXEORLSTXMSWHXBTBD7ZZG2Q, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:32.344389872 (43m48.125323238s) [noise=true , address=156.146.43.225:4448 , piece_id=YEHPZKKHQLJ57VOK6RENN4PFNJ6QYHN26Z3T47C4D6EM67Y4OOKA, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:33.777908892 (43m49.558842265s) [noise=true , address=79.127.219.47:4048 , piece_id=QRZOSWTJLWGVEAU22CQIVRAXL7Q36XXAKU5QUKEUC342HZCNSMVQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:36.839672883 (43m52.620606249s) [noise=true , address=64.49.13.138:4482 , piece_id=LRNWJIBYUO4LEXHXBOB6EDY7QVTLQHCBWF4Q4CL5PXTKSFT7WDIA, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:37.287385531 (43m53.068318901s) [noise=true , address=64.49.13.170:4721 , piece_id=RXL6GSXCBSTSPFRAEY3O73XGXXVDFGCLJEPDKPLJ3RJU3JGTES2A, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:37.761222869 (43m53.542156234s) [noise=true , address=79.127.205.246:4285 , piece_id=CHRRUNCJDF4HSPPBLJLKSZDR4XOJPQCWAPVWBZWH2E7DZXH7DXWQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:41.191286987 (43m56.972220365s) [noise=true , address=79.127.201.215:4359 , piece_id=ZNSGRGVJWNKHZPQG5BXNY7M6NBOX7J34BOFLJVLKFNPPSEOUKY4Q, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:36:26.095801764 (44m41.876735148s) [noise=true , address=77.90.31.16:4743 , piece_id=BHEZR3DMDDJ4LMFKTYSWFY2PJOGY4QBG3N2R43UAPYEANZSD7XLQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:30.656359404 (44m46.437292775s) [noise=true , address=77.90.31.18:4831 , piece_id=AZM43A6FZCVL57DNBZNYTV6H52XZNRHGLUULM2SXOVF3KEQ4USWQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: rpc: dial tcp 77.90.31.18:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:36:30.656440720 (44m46.4373741s) [noise=true , address=77.90.31.15:4723 , piece_id=XT32Y4DX554I3KVOCKS6NTIP3ASCQSAUPLV5NXG66WEGH7TENTNA, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: rpc: dial tcp 77.90.31.15:4723: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:36:30.656479718 (44m46.437413097s) [noise=true , address=77.90.31.19:4867 , piece_id=LNNOECSGUO4GYMOQPFMLX3SXDHKQCLAXKPEQNDWVDWVKZ5S4VOQA, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 159.223.143.39:60320->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:36:30.656518955 (44m46.437452334s) [noise=true , address=104.36.82.11:4138 , piece_id=PVYONMLKHQX6HGCM45ZTN2TAJMXOAM6PNURSBPRHNRF7FTUQI4TA, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: write tcp 159.223.143.39:42174->104.36.82.11:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:36:30.656571521 (44m46.43750491s) [seg_pos={0 0}, upload=675]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:36:30.686366243 (44m46.467299612s) [upload=675]: Done uploading segments... Jul 26 22:36:30.686414382 (44m46.467347759s) [upload=675]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.034665738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X........XXX.XXX.X.XX..X..X......X.............X..........] duration: 523.605603ms FAIL (2025-07-26 22:36:19 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 523.638438ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..................................X........X.X.X...XX.......] duration: 19.814772684s FAIL (2025-07-26 22:36:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.814813635s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 497.843122ms FAIL (2025-07-26 22:35:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 497.881438ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......XX..XXX..X..XX..X..XX.X....X....X........X...........] duration: 608.243877ms FAIL (2025-07-26 22:35:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 608.370789ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...................................X........X.X.X...XX......] duration: 19.370492542s FAIL (2025-07-26 22:35:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.370515065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 215.329677ms FAIL (2025-07-26 22:35:38 UTC) orbiter warning: uplink: too many requests Duration: 215.367725ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..X] duration: 1m0.041840454s FAIL (2025-07-26 22:35:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 661]: Got next segment. Inline: false Jul 26 22:34:27.951601133 (42m43.732534511s) [upload=661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:34:27.951620077 (42m43.732553458s) [upload=661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:34:27.987616768 (42m43.768550155s) [seg_pos={0 0}, upload=661]: Begin upload segment... Jul 26 22:34:27.987677035 (42m43.768610417s) [seg_pos={0 0}, upload=661]: Done begin upload segment. Jul 26 22:34:27.987710731 (42m43.76864416s) [noise=true , address=64.49.13.140:4516 , piece_id=VNEH2NXYNH4HC6UNLLIA4VI56ORP57CGDIVPR7BFONOLDIY7H6PA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.988347840 (42m43.769281227s) [noise=true , address=79.127.205.246:4288 , piece_id=VHIKAX2FZBGPSFRZMAWF6J27NHRBL6TSHE5FZMET6DXKXLWUUWUA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.988766163 (42m43.769699552s) [noise=true , address=79.127.219.51:4212 , piece_id=LYUAPOLRDGBWIDGMDX6LI4X2LZZP2QHOV2TOBEZHEGXKNZUZW3HA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.989132146 (42m43.770065535s) [noise=true , address=45.91.116.13:4175 , piece_id=ZLRYM2YSOEZ5NWSOWK5BPKYQIANMVJVSGP37DT64E4VUO7P6HTDQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.989479830 (42m43.770413219s) [noise=true , address=79.127.201.215:4347 , piece_id=XCKRUQYGDAUMHA22VBVP64UCZISA2GDA5PYW6LGFCB7I6577KXUQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.989836216 (42m43.770769605s) [noise=true , address=77.90.31.19:4871 , piece_id=QVCO3O6M37CI6BJOSOPAIAMIAUFXB2RGZNYPK3TBAZEADAKETM2A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.990228021 (42m43.77116141s) [noise=true , address=79.127.219.47:4069 , piece_id=4THURBUEKRP3JM7QZCWVRE5Y5ILP3GI26LA7X7QODSSJ4PNYY6EA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.993010821 (42m43.773944208s) [noise=true , address=77.90.31.17:4774 , piece_id=QG6LKYVX23N5L4CQFZ6IMRVOTLACEZUIOTBFOAZSLJFTEVLSRDIQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.993550663 (42m43.774484051s) [noise=true , address=77.90.31.15:4713 , piece_id=7ONT4B7UNK53T4LYXCAXK6NBUQGN45Z27DYZZYU4AI4QHDKRXS3A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.994855210 (42m43.7757886s) [noise=true , address=79.127.220.97:4361 , piece_id=2FH5LIZV2S5RAHNKOUOZMOGXA5VVGYIADMKQFTHSDYLFOVJNAD2Q, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.995342767 (42m43.776276154s) [noise=true , address=45.91.116.9:4267 , piece_id=2VHHVANAMTK6KINVQVVEPIUUTKJL3KJABZROESEE2SOLTMSBL3RQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.995778002 (42m43.776711391s) [noise=true , address=64.49.13.138:4482 , piece_id=NM5Z2MIKYTIP7XUEKOGUM4WVBYY5ZJZDMTSW4YV2BNHYA7VB6R7A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.996318448 (42m43.777251835s) [noise=true , address=79.127.219.38:4003 , piece_id=T3HRQHU4G34QQPLHXBXW5IMJTFKO2BWS4WSWRJKGO3M6D3FSW7ZA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.996711652 (42m43.777645038s) [noise=true , address=79.127.205.234:4612 , piece_id=PCVIO3LMYYN4TINAB5DVUS47NCHCRKHQOMQWMQIH26H4SEJQSC5A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.997100832 (42m43.778034226s) [noise=true , address=79.127.205.245:4237 , piece_id=GEJFAX5OPXASZESLJLRPOVE5W6LLUFHDP2KIO7FK4U37KY5T6FIA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.997489534 (42m43.778422924s) [noise=true , address=45.91.116.8:4065 , piece_id=X255M7QFAT5JOPBZO3DYBNHOKDKSSBODKMKXF7ZKVDSW7EW7TGHQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.997921939 (42m43.77885533s) [noise=true , address=77.90.31.12:4608 , piece_id=QCZE7AFBCTBJCNAVR67KQSU3FRNVNQQQTFP3LQ4QQRHBIKWCXJLQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.998345471 (42m43.77927886s) [noise=true , address=77.90.31.18:4817 , piece_id=NSAQ4EDSLCYZSSN7TB4HEQ5XXVVH6QSPHXJKIDI36BITOYYMVXXA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:28.056033124 (42m43.836966506s) [noise=true , address=79.127.219.47:4069 , piece_id=4THURBUEKRP3JM7QZCWVRE5Y5ILP3GI26LA7X7QODSSJ4PNYY6EA, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:18dK6RxbMxU4a46pW4YPiQW87xgpeoTb1ouqE9fzTdzTAFAa3W, address:79.127.219.47:4069): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:28.095674502 (42m43.876607888s) [upload=661]: Next returned nil segment Jul 26 22:34:28.095713720 (42m43.876647086s) [upload=661]: Waiting for error group managing segments... Jul 26 22:34:28.101217341 (42m43.882150718s) [noise=true , address=79.127.219.38:4003 , piece_id=T3HRQHU4G34QQPLHXBXW5IMJTFKO2BWS4WSWRJKGO3M6D3FSW7ZA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.119364618 (42m43.900298008s) [noise=true , address=79.127.219.51:4212 , piece_id=LYUAPOLRDGBWIDGMDX6LI4X2LZZP2QHOV2TOBEZHEGXKNZUZW3HA, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:28.138620488 (42m43.919553877s) [noise=true , address=79.127.220.97:4361 , piece_id=2FH5LIZV2S5RAHNKOUOZMOGXA5VVGYIADMKQFTHSDYLFOVJNAD2Q, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:28.144505433 (42m43.92543881s) [noise=true , address=77.90.31.17:4774 , piece_id=QG6LKYVX23N5L4CQFZ6IMRVOTLACEZUIOTBFOAZSLJFTEVLSRDIQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.146709992 (42m43.927643371s) [noise=true , address=77.90.31.12:4608 , piece_id=QCZE7AFBCTBJCNAVR67KQSU3FRNVNQQQTFP3LQ4QQRHBIKWCXJLQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.631921429 (42m44.412854796s) [noise=true , address=79.127.205.246:4288 , piece_id=VHIKAX2FZBGPSFRZMAWF6J27NHRBL6TSHE5FZMET6DXKXLWUUWUA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.678668075 (42m44.459601453s) [noise=true , address=79.127.205.234:4612 , piece_id=PCVIO3LMYYN4TINAB5DVUS47NCHCRKHQOMQWMQIH26H4SEJQSC5A, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:29.137248803 (42m44.918182166s) [noise=true , address=79.127.205.245:4237 , piece_id=GEJFAX5OPXASZESLJLRPOVE5W6LLUFHDP2KIO7FK4U37KY5T6FIA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:31.771346822 (42m47.552280203s) [noise=true , address=45.91.116.13:4175 , piece_id=ZLRYM2YSOEZ5NWSOWK5BPKYQIANMVJVSGP37DT64E4VUO7P6HTDQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:32.565641205 (42m48.346574594s) [noise=true , address=45.91.116.8:4065 , piece_id=X255M7QFAT5JOPBZO3DYBNHOKDKSSBODKMKXF7ZKVDSW7EW7TGHQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:17EPCQmYb816aegE9vPSoJYByMUEvhP4DUiabis2gh9QjXQC3s, address:45.91.116.8:4065): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:33.360957537 (42m49.141890915s) [noise=true , address=79.127.201.215:4347 , piece_id=XCKRUQYGDAUMHA22VBVP64UCZISA2GDA5PYW6LGFCB7I6577KXUQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:34.784741480 (42m50.565674868s) [noise=true , address=45.91.116.9:4267 , piece_id=2VHHVANAMTK6KINVQVVEPIUUTKJL3KJABZROESEE2SOLTMSBL3RQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:35.843967454 (42m51.624900832s) [noise=true , address=64.49.13.138:4482 , piece_id=NM5Z2MIKYTIP7XUEKOGUM4WVBYY5ZJZDMTSW4YV2BNHYA7VB6R7A, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:36.075665804 (42m51.85659917s) [noise=true , address=64.49.13.140:4516 , piece_id=VNEH2NXYNH4HC6UNLLIA4VI56ORP57CGDIVPR7BFONOLDIY7H6PA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:35:27.909515279 (43m43.690448663s) [noise=true , address=77.90.31.19:4871 , piece_id=QVCO3O6M37CI6BJOSOPAIAMIAUFXB2RGZNYPK3TBAZEADAKETM2A, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:35:27.909647421 (43m43.690580801s) [noise=true , address=77.90.31.15:4713 , piece_id=7ONT4B7UNK53T4LYXCAXK6NBUQGN45Z27DYZZYU4AI4QHDKRXS3A, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: write tcp 159.223.143.39:53460->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:35:27.909740858 (43m43.690674235s) [noise=true , address=77.90.31.18:4817 , piece_id=NSAQ4EDSLCYZSSN7TB4HEQ5XXVVH6QSPHXJKIDI36BITOYYMVXXA, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: write tcp 159.223.143.39:56042->77.90.31.18:4817: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:35:27.909776921 (43m43.690710298s) [seg_pos={0 0}, upload=661]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:35:27.947311027 (43m43.728244408s) [upload=661]: Done uploading segments... Jul 26 22:35:27.947342152 (43m43.72827553s) [upload=661]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.041877536s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 489.097538ms FAIL (2025-07-26 22:35:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.131552ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..X........XXX.XXX.X.XX..X..X......X.............X.........] duration: 441.152948ms FAIL (2025-07-26 22:35:16 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 441.184713ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 357.16577ms FAIL (2025-07-26 22:34:53 UTC) orbiter warning: uplink: too many requests Duration: 357.196019ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [........XX..XXX..X..XX..X..XX.X....X....X........X..........] duration: 417.635535ms FAIL (2025-07-26 22:34:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.78023ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 522.967641ms FAIL (2025-07-26 22:34:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 522.990313ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....................................X........X.X.X...XX.....] duration: 1.53728997s FAIL (2025-07-26 22:34:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.538325747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 750.789941ms FAIL (2025-07-26 22:34:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 750.847867ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..X........XXX.XXX.X.XX..X..X......X.............X........] duration: 493.470179ms FAIL (2025-07-26 22:34:12 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 493.499066ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)