Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XXX.X.XX..X..X......X.............X................X.......] duration: 458.100294ms FAIL (2025-07-26 22:47:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.129417ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...................X........X.X.X...XX.................X.X..] duration: 7.793275492s FAIL (2025-07-26 22:47:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.793314608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 483.737491ms FAIL (2025-07-26 22:47:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.764716ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.] duration: 22.104376852s FAIL (2025-07-26 22:47:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== : Starting upload Jul 26 22:45:46.202966629 (54m1.983899996s) [upload=833]: Done starting upload Jul 26 22:45:46.203186109 (54m1.984119479s) [upload=833]: Uploading segments... Jul 26 22:45:46.204797212 (54m1.985730595s) [upload=833]: Got next segment. Inline: false Jul 26 22:45:46.204889895 (54m1.985823275s) [upload=833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:45:46.204896827 (54m1.985830207s) [upload=833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:45:46.245793659 (54m2.02672705s) [seg_pos={0 0}, upload=833]: Begin upload segment... Jul 26 22:45:46.245858437 (54m2.026791818s) [seg_pos={0 0}, upload=833]: Done begin upload segment. Jul 26 22:45:46.245892262 (54m2.026825649s) [noise=true , address=79.127.219.51:4210 , piece_id=V4ZCDPTNC2OJFMRQ6UAD3MJZKOVN6YTLNXOYXQIUQWPXYNR2FUWA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.246442503 (54m2.027375892s) [noise=true , address=79.127.205.245:4239 , piece_id=DFOVVPY37T7ZWRIG2MOONBQLWKO3ZFQKWJQWNNAIMLEM2S4QDDYQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.246887839 (54m2.027821228s) [noise=true , address=79.127.219.47:4061 , piece_id=BKY3CLW6LCFW467BSB57QWTPXQ4FENKZGIUQ522KDLBK5KEXWWEA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.247232391 (54m2.028165783s) [noise=true , address=77.90.31.19:4867 , piece_id=P4JSUDFVMZSXIIPXQFA6CRTYNEE5QHVMJRJJ2CIMX5HJMCHGOJJA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.247846641 (54m2.028780028s) [noise=true , address=79.127.201.215:4349 , piece_id=A3TQVCLGTTYSSZY6JNDKOMVIE6O2EBSVTBSPOJSNAW6WEWFQC32A, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.248479097 (54m2.029412486s) [noise=true , address=79.127.205.233:4623 , piece_id=3FNY6K37CA5BUP7D5HW4K6LHGYET74D67OHX4IDCI5DZHNABP63A, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.249010085 (54m2.029943471s) [noise=true , address=77.90.31.16:4748 , piece_id=VLZRBTVOVTZRDCV4XQVVL2UFYUNBDRD5OLHRHREKW4QKDKYWTNZA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.249509407 (54m2.030442794s) [noise=true , address=45.91.116.9:4267 , piece_id=DMHK6YW57JMQDJ34P7LDGGYVFHTEFV6GODLQY7CSYIDFBBBZAYXQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.250006087 (54m2.030939474s) [noise=true , address=79.127.219.47:4048 , piece_id=QDZGNOBKRLPIME3VSD23JT4KZPO35KREXCUTXJJ6RS6CVEOFSWMQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.250528343 (54m2.03146173s) [noise=true , address=45.91.116.14:4206 , piece_id=XHDKZ7WROIGGSC4SA4NINNKJIHXGRSEI4APDCEKWPXPQIV6XBTDA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.251046173 (54m2.031979556s) [noise=true , address=79.127.219.38:4004 , piece_id=HP434MWG5F3MHYOOJBIJ6WAE3OTJ5JL4OBW3BSM36Q4OKCNAXGOQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.251513861 (54m2.03244729s) [noise=true , address=156.146.43.225:4434 , piece_id=6W24KWWFV5RSHPIYGUPIWXVLMEUPAXPZ4GOBTKHZZLQH3EFBKBMA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.251939466 (54m2.032872854s) [noise=true , address=77.90.31.10:4533 , piece_id=P46CZ62WQVGXN53QLX2R3T2OD34SPKQGP3U3LLRIMBQEOBJBJHQQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.252413827 (54m2.033347214s) [noise=true , address=77.90.31.21:4902 , piece_id=4567A4ZGHK2UNWDJGCOO6FNZJWX7EQ74RTZHHYU7DXSEHMG3JSPA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.252849468 (54m2.033782853s) [noise=true , address=79.127.220.101:4048 , piece_id=URZ7O6A5HHB7IRIU4GKTXA6XZ5CI4PHDHDYHWXTB3DAXETQKESKA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.253238338 (54m2.034171722s) [noise=true , address=156.146.43.225:4449 , piece_id=EWLND7ROHPKIJ2WYPIV7VWCYSDYRFNIN3ZS67AMBWGM63SOX3LMA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.260547504 (54m2.041480894s) [noise=true , address=64.49.13.155:4923 , piece_id=SQIRFYBVQIEGQAKZS5KPMJH575AGCBAALDUMO44GMNT7YBJF6KDQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.261085468 (54m2.042018855s) [noise=true , address=79.127.219.38:4009 , piece_id=CONFXHDMJBNV7O7K2M4UOIXLW64KT6O5GRTUWHECIPHJS455J6XA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.423745889 (54m2.204679267s) [noise=true , address=79.127.219.47:4061 , piece_id=BKY3CLW6LCFW467BSB57QWTPXQ4FENKZGIUQ522KDLBK5KEXWWEA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:46.443106652 (54m2.224040049s) [upload=833]: Next returned nil segment Jul 26 22:45:46.443130916 (54m2.224064301s) [upload=833]: Waiting for error group managing segments... Jul 26 22:45:46.447211320 (54m2.228144708s) [noise=true , address=156.146.43.225:4434 , piece_id=6W24KWWFV5RSHPIYGUPIWXVLMEUPAXPZ4GOBTKHZZLQH3EFBKBMA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:46.448876892 (54m2.229810265s) [noise=true , address=79.127.219.38:4004 , piece_id=HP434MWG5F3MHYOOJBIJ6WAE3OTJ5JL4OBW3BSM36Q4OKCNAXGOQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:46.452978238 (54m2.233911616s) [noise=true , address=156.146.43.225:4449 , piece_id=EWLND7ROHPKIJ2WYPIV7VWCYSDYRFNIN3ZS67AMBWGM63SOX3LMA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:46.454984612 (54m2.235917991s) [noise=true , address=45.91.116.14:4206 , piece_id=XHDKZ7WROIGGSC4SA4NINNKJIHXGRSEI4APDCEKWPXPQIV6XBTDA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:46.459121644 (54m2.240055012s) [noise=true , address=79.127.219.38:4009 , piece_id=CONFXHDMJBNV7O7K2M4UOIXLW64KT6O5GRTUWHECIPHJS455J6XA, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:46.486062323 (54m2.266995694s) [noise=true , address=77.90.31.21:4902 , piece_id=4567A4ZGHK2UNWDJGCOO6FNZJWX7EQ74RTZHHYU7DXSEHMG3JSPA, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:46.608692694 (54m2.389626077s) [noise=true , address=79.127.220.101:4048 , piece_id=URZ7O6A5HHB7IRIU4GKTXA6XZ5CI4PHDHDYHWXTB3DAXETQKESKA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:47.112432137 (54m2.893365513s) [noise=true , address=79.127.205.245:4239 , piece_id=DFOVVPY37T7ZWRIG2MOONBQLWKO3ZFQKWJQWNNAIMLEM2S4QDDYQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:47.186473067 (54m2.967406437s) [noise=true , address=79.127.205.233:4623 , piece_id=3FNY6K37CA5BUP7D5HW4K6LHGYET74D67OHX4IDCI5DZHNABP63A, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:47.625907094 (54m3.406840472s) [noise=true , address=64.49.13.155:4923 , piece_id=SQIRFYBVQIEGQAKZS5KPMJH575AGCBAALDUMO44GMNT7YBJF6KDQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:48.465067861 (54m4.246001248s) [noise=true , address=77.90.31.19:4867 , piece_id=P4JSUDFVMZSXIIPXQFA6CRTYNEE5QHVMJRJJ2CIMX5HJMCHGOJJA, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:49.118436890 (54m4.899370254s) [noise=true , address=79.127.219.51:4210 , piece_id=V4ZCDPTNC2OJFMRQ6UAD3MJZKOVN6YTLNXOYXQIUQWPXYNR2FUWA, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:49.423380706 (54m5.204314076s) [noise=true , address=79.127.219.47:4048 , piece_id=QDZGNOBKRLPIME3VSD23JT4KZPO35KREXCUTXJJ6RS6CVEOFSWMQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:50.323690175 (54m6.10462355s) [noise=true , address=45.91.116.9:4267 , piece_id=DMHK6YW57JMQDJ34P7LDGGYVFHTEFV6GODLQY7CSYIDFBBBZAYXQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:04.604842454 (54m20.385775821s) [noise=true , address=79.127.201.215:4349 , piece_id=A3TQVCLGTTYSSZY6JNDKOMVIE6O2EBSVTBSPOJSNAW6WEWFQC32A, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:46:46.164497311 (55m1.945430675s) [noise=true , address=77.90.31.10:4533 , piece_id=P46CZ62WQVGXN53QLX2R3T2OD34SPKQGP3U3LLRIMBQEOBJBJHQQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: rpc: dial tcp 77.90.31.10:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:46:46.164817459 (55m1.945750822s) [noise=true , address=77.90.31.16:4748 , piece_id=VLZRBTVOVTZRDCV4XQVVL2UFYUNBDRD5OLHRHREKW4QKDKYWTNZA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:46:46.164865717 (55m1.945799101s) [seg_pos={0 0}, upload=833]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:46:46.194508974 (55m1.975442368s) [upload=833]: Done uploading segments... Jul 26 22:46:46.194542935 (55m1.975476318s) [upload=833]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 22.104400806s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 556.641135ms FAIL (2025-07-26 22:47:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 556.687279ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 194.856863ms FAIL (2025-07-26 22:47:01 UTC) orbiter warning: uplink: too many requests Duration: 194.92742ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.XXX.X.XX..X..X......X.............X................X......] duration: 422.186735ms FAIL (2025-07-26 22:46:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.224116ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....................X........X.X.X...XX.................X.X.] duration: 11.938166154s FAIL (2025-07-26 22:46:54 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.938195765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X..XX..X..XX.X....X....X........X..........................] duration: 469.003909ms FAIL (2025-07-26 22:46:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 469.038369ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X] duration: 1m0.03311186s FAIL (2025-07-26 22:46:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== : Starting upload Jul 26 22:45:46.202966629 (54m1.983899996s) [upload=833]: Done starting upload Jul 26 22:45:46.203186109 (54m1.984119479s) [upload=833]: Uploading segments... Jul 26 22:45:46.204797212 (54m1.985730595s) [upload=833]: Got next segment. Inline: false Jul 26 22:45:46.204889895 (54m1.985823275s) [upload=833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:45:46.204896827 (54m1.985830207s) [upload=833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:45:46.245793659 (54m2.02672705s) [seg_pos={0 0}, upload=833]: Begin upload segment... Jul 26 22:45:46.245858437 (54m2.026791818s) [seg_pos={0 0}, upload=833]: Done begin upload segment. Jul 26 22:45:46.245892262 (54m2.026825649s) [noise=true , address=79.127.219.51:4210 , piece_id=V4ZCDPTNC2OJFMRQ6UAD3MJZKOVN6YTLNXOYXQIUQWPXYNR2FUWA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.246442503 (54m2.027375892s) [noise=true , address=79.127.205.245:4239 , piece_id=DFOVVPY37T7ZWRIG2MOONBQLWKO3ZFQKWJQWNNAIMLEM2S4QDDYQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.246887839 (54m2.027821228s) [noise=true , address=79.127.219.47:4061 , piece_id=BKY3CLW6LCFW467BSB57QWTPXQ4FENKZGIUQ522KDLBK5KEXWWEA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.247232391 (54m2.028165783s) [noise=true , address=77.90.31.19:4867 , piece_id=P4JSUDFVMZSXIIPXQFA6CRTYNEE5QHVMJRJJ2CIMX5HJMCHGOJJA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.247846641 (54m2.028780028s) [noise=true , address=79.127.201.215:4349 , piece_id=A3TQVCLGTTYSSZY6JNDKOMVIE6O2EBSVTBSPOJSNAW6WEWFQC32A, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.248479097 (54m2.029412486s) [noise=true , address=79.127.205.233:4623 , piece_id=3FNY6K37CA5BUP7D5HW4K6LHGYET74D67OHX4IDCI5DZHNABP63A, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.249010085 (54m2.029943471s) [noise=true , address=77.90.31.16:4748 , piece_id=VLZRBTVOVTZRDCV4XQVVL2UFYUNBDRD5OLHRHREKW4QKDKYWTNZA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.249509407 (54m2.030442794s) [noise=true , address=45.91.116.9:4267 , piece_id=DMHK6YW57JMQDJ34P7LDGGYVFHTEFV6GODLQY7CSYIDFBBBZAYXQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.250006087 (54m2.030939474s) [noise=true , address=79.127.219.47:4048 , piece_id=QDZGNOBKRLPIME3VSD23JT4KZPO35KREXCUTXJJ6RS6CVEOFSWMQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.250528343 (54m2.03146173s) [noise=true , address=45.91.116.14:4206 , piece_id=XHDKZ7WROIGGSC4SA4NINNKJIHXGRSEI4APDCEKWPXPQIV6XBTDA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.251046173 (54m2.031979556s) [noise=true , address=79.127.219.38:4004 , piece_id=HP434MWG5F3MHYOOJBIJ6WAE3OTJ5JL4OBW3BSM36Q4OKCNAXGOQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.251513861 (54m2.03244729s) [noise=true , address=156.146.43.225:4434 , piece_id=6W24KWWFV5RSHPIYGUPIWXVLMEUPAXPZ4GOBTKHZZLQH3EFBKBMA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.251939466 (54m2.032872854s) [noise=true , address=77.90.31.10:4533 , piece_id=P46CZ62WQVGXN53QLX2R3T2OD34SPKQGP3U3LLRIMBQEOBJBJHQQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.252413827 (54m2.033347214s) [noise=true , address=77.90.31.21:4902 , piece_id=4567A4ZGHK2UNWDJGCOO6FNZJWX7EQ74RTZHHYU7DXSEHMG3JSPA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.252849468 (54m2.033782853s) [noise=true , address=79.127.220.101:4048 , piece_id=URZ7O6A5HHB7IRIU4GKTXA6XZ5CI4PHDHDYHWXTB3DAXETQKESKA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.253238338 (54m2.034171722s) [noise=true , address=156.146.43.225:4449 , piece_id=EWLND7ROHPKIJ2WYPIV7VWCYSDYRFNIN3ZS67AMBWGM63SOX3LMA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.260547504 (54m2.041480894s) [noise=true , address=64.49.13.155:4923 , piece_id=SQIRFYBVQIEGQAKZS5KPMJH575AGCBAALDUMO44GMNT7YBJF6KDQ, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.261085468 (54m2.042018855s) [noise=true , address=79.127.219.38:4009 , piece_id=CONFXHDMJBNV7O7K2M4UOIXLW64KT6O5GRTUWHECIPHJS455J6XA, seg_pos={0 0}, upload=833]: Uploading piece... Jul 26 22:45:46.423745889 (54m2.204679267s) [noise=true , address=79.127.219.47:4061 , piece_id=BKY3CLW6LCFW467BSB57QWTPXQ4FENKZGIUQ522KDLBK5KEXWWEA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:46.443106652 (54m2.224040049s) [upload=833]: Next returned nil segment Jul 26 22:45:46.443130916 (54m2.224064301s) [upload=833]: Waiting for error group managing segments... Jul 26 22:45:46.447211320 (54m2.228144708s) [noise=true , address=156.146.43.225:4434 , piece_id=6W24KWWFV5RSHPIYGUPIWXVLMEUPAXPZ4GOBTKHZZLQH3EFBKBMA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:46.448876892 (54m2.229810265s) [noise=true , address=79.127.219.38:4004 , piece_id=HP434MWG5F3MHYOOJBIJ6WAE3OTJ5JL4OBW3BSM36Q4OKCNAXGOQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:46.452978238 (54m2.233911616s) [noise=true , address=156.146.43.225:4449 , piece_id=EWLND7ROHPKIJ2WYPIV7VWCYSDYRFNIN3ZS67AMBWGM63SOX3LMA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:46.454984612 (54m2.235917991s) [noise=true , address=45.91.116.14:4206 , piece_id=XHDKZ7WROIGGSC4SA4NINNKJIHXGRSEI4APDCEKWPXPQIV6XBTDA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:46.459121644 (54m2.240055012s) [noise=true , address=79.127.219.38:4009 , piece_id=CONFXHDMJBNV7O7K2M4UOIXLW64KT6O5GRTUWHECIPHJS455J6XA, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:46.486062323 (54m2.266995694s) [noise=true , address=77.90.31.21:4902 , piece_id=4567A4ZGHK2UNWDJGCOO6FNZJWX7EQ74RTZHHYU7DXSEHMG3JSPA, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:46.608692694 (54m2.389626077s) [noise=true , address=79.127.220.101:4048 , piece_id=URZ7O6A5HHB7IRIU4GKTXA6XZ5CI4PHDHDYHWXTB3DAXETQKESKA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:47.112432137 (54m2.893365513s) [noise=true , address=79.127.205.245:4239 , piece_id=DFOVVPY37T7ZWRIG2MOONBQLWKO3ZFQKWJQWNNAIMLEM2S4QDDYQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:47.186473067 (54m2.967406437s) [noise=true , address=79.127.205.233:4623 , piece_id=3FNY6K37CA5BUP7D5HW4K6LHGYET74D67OHX4IDCI5DZHNABP63A, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:47.625907094 (54m3.406840472s) [noise=true , address=64.49.13.155:4923 , piece_id=SQIRFYBVQIEGQAKZS5KPMJH575AGCBAALDUMO44GMNT7YBJF6KDQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:48.465067861 (54m4.246001248s) [noise=true , address=77.90.31.19:4867 , piece_id=P4JSUDFVMZSXIIPXQFA6CRTYNEE5QHVMJRJJ2CIMX5HJMCHGOJJA, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:49.118436890 (54m4.899370254s) [noise=true , address=79.127.219.51:4210 , piece_id=V4ZCDPTNC2OJFMRQ6UAD3MJZKOVN6YTLNXOYXQIUQWPXYNR2FUWA, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:49.423380706 (54m5.204314076s) [noise=true , address=79.127.219.47:4048 , piece_id=QDZGNOBKRLPIME3VSD23JT4KZPO35KREXCUTXJJ6RS6CVEOFSWMQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:45:50.323690175 (54m6.10462355s) [noise=true , address=45.91.116.9:4267 , piece_id=DMHK6YW57JMQDJ34P7LDGGYVFHTEFV6GODLQY7CSYIDFBBBZAYXQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:04.604842454 (54m20.385775821s) [noise=true , address=79.127.201.215:4349 , piece_id=A3TQVCLGTTYSSZY6JNDKOMVIE6O2EBSVTBSPOJSNAW6WEWFQC32A, seg_pos={0 0}, upload=833]: Done uploading piece. err: <nil> Jul 26 22:46:46.164497311 (55m1.945430675s) [noise=true , address=77.90.31.10:4533 , piece_id=P46CZ62WQVGXN53QLX2R3T2OD34SPKQGP3U3LLRIMBQEOBJBJHQQ, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: rpc: dial tcp 77.90.31.10:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:46:46.164817459 (55m1.945750822s) [noise=true , address=77.90.31.16:4748 , piece_id=VLZRBTVOVTZRDCV4XQVVL2UFYUNBDRD5OLHRHREKW4QKDKYWTNZA, seg_pos={0 0}, upload=833]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:46:46.164865717 (55m1.945799101s) [seg_pos={0 0}, upload=833]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:46:46.194508974 (55m1.975442368s) [upload=833]: Done uploading segments... Jul 26 22:46:46.194542935 (55m1.975476318s) [upload=833]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.033142695s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....................X........X.X.X...XX.................X.X] duration: 1m0.001172224s FAIL (2025-07-26 22:46:42 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001199738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 453.397743ms FAIL (2025-07-26 22:46:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.477447ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 217.11117ms FAIL (2025-07-26 22:45:52 UTC) orbiter warning: uplink: too many requests Duration: 217.165359ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X..XX..X..XX.X....X....X........X.........................] duration: 405.164825ms FAIL (2025-07-26 22:45:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 405.196024ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX.XXX.X.XX..X..X......X.............X................X.....] duration: 552.620987ms FAIL (2025-07-26 22:45:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 552.644401ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [......................X........X.X.X...XX.................X.] duration: 11.859107262s FAIL (2025-07-26 22:45:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.859127605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.......................X........X.X.X...XX.................X] duration: 1m0.001060063s FAIL (2025-07-26 22:45:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001095582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 199.677062ms FAIL (2025-07-26 22:45:22 UTC) orbiter warning: uplink: too many requests Duration: 199.787063ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 496.308272ms FAIL (2025-07-26 22:45:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.347318ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..X..XX..X..XX.X....X....X........X........................] duration: 637.358141ms FAIL (2025-07-26 22:45:09 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 637.390894ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXX.XXX.X.XX..X..X......X.............X................X....] duration: 375.612162ms FAIL (2025-07-26 22:44:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 375.639533ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.] duration: 6.264211583s FAIL (2025-07-26 22:44:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 26 22:43:41.110295795 (51m56.891229164s) [upload=788]: Done uploading segments... Jul 26 22:43:41.110322252 (51m56.891255627s) [upload=788]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:43:41.194853734 (51m56.975787099s) [upload=802]: Starting upload Jul 26 22:43:41.194948018 (51m56.975881403s) [upload=802]: Done starting upload Jul 26 22:43:41.195214873 (51m56.97614824s) [upload=802]: Uploading segments... Jul 26 22:43:41.195440638 (51m56.976374016s) [upload=802]: Got next segment. Inline: false Jul 26 22:43:41.195523504 (51m56.976456868s) [upload=802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:41.195529671 (51m56.976463035s) [upload=802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:41.231947842 (51m57.012881229s) [seg_pos={0 0}, upload=802]: Begin upload segment... Jul 26 22:43:41.232018620 (51m57.012952006s) [seg_pos={0 0}, upload=802]: Done begin upload segment. Jul 26 22:43:41.232103172 (51m57.013036547s) [noise=true , address=77.90.31.19:4874 , piece_id=RLVLQGUYAJRH53SBRIA25EE6L3YCXCHIN7CQ22MGFE5QJZECJYNQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232134530 (51m57.013067913s) [noise=true , address=79.127.205.245:4228 , piece_id=IIVPND2UAM7VD6DA4HGKOX3B3ZVHNYP63ZPWNMN3F47UVZ57TD6A, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232598891 (51m57.013532278s) [noise=true , address=104.36.82.35:4607 , piece_id=X36Y42PQULA5A4B6UZMCPQK34FM4INNMUMQHKKZL2FG6MOELOJFA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232997403 (51m57.013930793s) [noise=true , address=64.49.13.138:4482 , piece_id=HTUBWJUH7EUYOMFWBLQX25WAJGRJCH3DGX2V6HAZSJ2IUMEI3LCQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.233811231 (51m57.014744615s) [noise=true , address=64.49.13.137:4500 , piece_id=QJKNHVY5IRULGIELVRDNXQMWI4C6XLPLKH5QX3OLEG4IEOIN4MZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234107016 (51m57.0150404s) [noise=true , address=79.127.219.48:4090 , piece_id=SMYUBZNSBP4OWQZFEIZ5GIZ6QRYLEKXJI5EFDKHSZJID744JTTDA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234225439 (51m57.015158817s) [noise=true , address=79.127.219.51:4210 , piece_id=N5X4TBNI5MBOJKBING7YL55LVQU6B5ZQBSGAM3YQFTVACO4BSC5Q, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234530128 (51m57.01546354s) [noise=true , address=104.36.82.10:4101 , piece_id=S6RT7PW6TVY7K2KNB42YKXEQMDTSMOFJK4CDNO3MKB6FDXLVYBXA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234677395 (51m57.015610776s) [noise=true , address=77.90.31.18:4816 , piece_id=HY55JOE2FS3B3EGYJ3C43MTAXQ33EBQKUUX27SPJ4IMRFVDDOOZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235151053 (51m57.016084431s) [noise=true , address=79.127.220.101:4048 , piece_id=UUV32R7XFPCNDEF43ZOMWC67JJZ7U7YDMLXFXAHTIXUKESNCB4MA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235529599 (51m57.016462976s) [noise=true , address=156.146.43.225:4449 , piece_id=G44WXWCQDQOSSOBG64LRAU3DLOUSH2ULL4I3GT6THFAGYD6BQRTA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235990344 (51m57.016923728s) [noise=true , address=79.127.219.38:4004 , piece_id=JPDFRQ2YEH3KF6RGT4QIE47C6ZI6HMHBTUOXTHGRYZ4ZNDMFEZJQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.236414748 (51m57.017348125s) [noise=true , address=79.127.205.246:4282 , piece_id=HC27CTDAG7TRJ5YQQFFMG7SFDRIN3JRTMFUI42KMK35T3TIB4TZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.237577245 (51m57.018510629s) [noise=true , address=104.36.82.7:4918 , piece_id=3KJZTIAZBP7WBZMHYTYYO2XWUBANMLBT7KUCUKTYHSJ2MZRKXQAA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.238217941 (51m57.019151322s) [noise=true , address=79.127.205.233:4633 , piece_id=VZE6UZ4SW2XBZRNZMBP4TTDDVBRQGYM6SQKYR5BPYBV446K5KPGA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.238950587 (51m57.019883971s) [noise=true , address=77.90.31.11:4571 , piece_id=5XMT4IU4WYZNCLUHFKE2ZYDJTGIX6ESHWWHTNOW22IUB3GE5V3JQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.239460727 (51m57.020394109s) [noise=true , address=77.90.31.10:4538 , piece_id=Z6GPO6B4JMM27B5CJE64GZMAEDWAJKAAVJBRCPRAB5ULHMI7XLFQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.239899361 (51m57.020832748s) [noise=true , address=79.127.205.234:4616 , piece_id=XQSAFSV4ZC5ACSFVVWKAL7BPLQ2KVT2ZEYU6LUM3Q3EFOP3S22PQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.304182241 (51m57.085115671s) [upload=802]: Next returned nil segment Jul 26 22:43:41.304206569 (51m57.085139943s) [upload=802]: Waiting for error group managing segments... Jul 26 22:43:41.309312769 (51m57.090246201s) [noise=true , address=79.127.219.38:4004 , piece_id=JPDFRQ2YEH3KF6RGT4QIE47C6ZI6HMHBTUOXTHGRYZ4ZNDMFEZJQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.366028377 (51m57.146961744s) [noise=true , address=77.90.31.11:4571 , piece_id=5XMT4IU4WYZNCLUHFKE2ZYDJTGIX6ESHWWHTNOW22IUB3GE5V3JQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.385684427 (51m57.166617806s) [noise=true , address=79.127.219.51:4210 , piece_id=N5X4TBNI5MBOJKBING7YL55LVQU6B5ZQBSGAM3YQFTVACO4BSC5Q, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:41.823755988 (51m57.604689367s) [noise=true , address=79.127.205.246:4282 , piece_id=HC27CTDAG7TRJ5YQQFFMG7SFDRIN3JRTMFUI42KMK35T3TIB4TZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.837645590 (51m57.61857896s) [noise=true , address=79.127.205.245:4228 , piece_id=IIVPND2UAM7VD6DA4HGKOX3B3ZVHNYP63ZPWNMN3F47UVZ57TD6A, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.944332236 (51m57.725265612s) [noise=true , address=64.49.13.138:4482 , piece_id=HTUBWJUH7EUYOMFWBLQX25WAJGRJCH3DGX2V6HAZSJ2IUMEI3LCQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:42.082424997 (51m57.86335838s) [noise=true , address=79.127.205.234:4616 , piece_id=XQSAFSV4ZC5ACSFVVWKAL7BPLQ2KVT2ZEYU6LUM3Q3EFOP3S22PQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:42.339902702 (51m58.120836064s) [noise=true , address=79.127.205.233:4633 , piece_id=VZE6UZ4SW2XBZRNZMBP4TTDDVBRQGYM6SQKYR5BPYBV446K5KPGA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:42.616942028 (51m58.397875405s) [noise=true , address=79.127.219.48:4090 , piece_id=SMYUBZNSBP4OWQZFEIZ5GIZ6QRYLEKXJI5EFDKHSZJID744JTTDA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:43.761718655 (51m59.542652024s) [noise=true , address=156.146.43.225:4449 , piece_id=G44WXWCQDQOSSOBG64LRAU3DLOUSH2ULL4I3GT6THFAGYD6BQRTA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:44.144083824 (51m59.925017201s) [noise=true , address=64.49.13.137:4500 , piece_id=QJKNHVY5IRULGIELVRDNXQMWI4C6XLPLKH5QX3OLEG4IEOIN4MZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:45.274140250 (52m1.055073612s) [noise=true , address=79.127.220.101:4048 , piece_id=UUV32R7XFPCNDEF43ZOMWC67JJZ7U7YDMLXFXAHTIXUKESNCB4MA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:46.921015676 (52m2.70194904s) [noise=true , address=77.90.31.19:4874 , piece_id=RLVLQGUYAJRH53SBRIA25EE6L3YCXCHIN7CQ22MGFE5QJZECJYNQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:52.019885342 (52m7.800818718s) [noise=true , address=77.90.31.10:4538 , piece_id=Z6GPO6B4JMM27B5CJE64GZMAEDWAJKAAVJBRCPRAB5ULHMI7XLFQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:59.877267403 (52m15.658200785s) [noise=true , address=77.90.31.18:4816 , piece_id=HY55JOE2FS3B3EGYJ3C43MTAXQ33EBQKUUX27SPJ4IMRFVDDOOZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:44:41.166057950 (52m56.946991332s) [noise=true , address=104.36.82.35:4607 , piece_id=X36Y42PQULA5A4B6UZMCPQK34FM4INNMUMQHKKZL2FG6MOELOJFA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1Ex8eUvL1Hqxd5EffckSvBeZDpGrMGhKENyCPvqHybtQf3YQzM, address:104.36.82.35:4607): protocol: rpc: dial tcp 104.36.82.35:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166171721 (52m56.9471051s) [noise=true , address=104.36.82.10:4101 , piece_id=S6RT7PW6TVY7K2KNB42YKXEQMDTSMOFJK4CDNO3MKB6FDXLVYBXA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166252938 (52m56.947186315s) [noise=true , address=104.36.82.7:4918 , piece_id=3KJZTIAZBP7WBZMHYTYYO2XWUBANMLBT7KUCUKTYHSJ2MZRKXQAA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:12EMGZ4hpXmBp4TwemLrbKAVtjBmtexMzEahRTzFw42qQipsAG, address:104.36.82.7:4918): protocol: write tcp 159.223.143.39:37030->104.36.82.7:4918: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166278974 (52m56.947212365s) [seg_pos={0 0}, upload=802]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:44:41.189117481 (52m56.97005085s) [upload=802]: Done uploading segments... Jul 26 22:44:41.189141942 (52m56.970075317s) [upload=802]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.264232796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX] duration: 1m0.025681074s FAIL (2025-07-26 22:44:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 26 22:43:41.110295795 (51m56.891229164s) [upload=788]: Done uploading segments... Jul 26 22:43:41.110322252 (51m56.891255627s) [upload=788]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:43:41.194853734 (51m56.975787099s) [upload=802]: Starting upload Jul 26 22:43:41.194948018 (51m56.975881403s) [upload=802]: Done starting upload Jul 26 22:43:41.195214873 (51m56.97614824s) [upload=802]: Uploading segments... Jul 26 22:43:41.195440638 (51m56.976374016s) [upload=802]: Got next segment. Inline: false Jul 26 22:43:41.195523504 (51m56.976456868s) [upload=802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:41.195529671 (51m56.976463035s) [upload=802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:41.231947842 (51m57.012881229s) [seg_pos={0 0}, upload=802]: Begin upload segment... Jul 26 22:43:41.232018620 (51m57.012952006s) [seg_pos={0 0}, upload=802]: Done begin upload segment. Jul 26 22:43:41.232103172 (51m57.013036547s) [noise=true , address=77.90.31.19:4874 , piece_id=RLVLQGUYAJRH53SBRIA25EE6L3YCXCHIN7CQ22MGFE5QJZECJYNQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232134530 (51m57.013067913s) [noise=true , address=79.127.205.245:4228 , piece_id=IIVPND2UAM7VD6DA4HGKOX3B3ZVHNYP63ZPWNMN3F47UVZ57TD6A, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232598891 (51m57.013532278s) [noise=true , address=104.36.82.35:4607 , piece_id=X36Y42PQULA5A4B6UZMCPQK34FM4INNMUMQHKKZL2FG6MOELOJFA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232997403 (51m57.013930793s) [noise=true , address=64.49.13.138:4482 , piece_id=HTUBWJUH7EUYOMFWBLQX25WAJGRJCH3DGX2V6HAZSJ2IUMEI3LCQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.233811231 (51m57.014744615s) [noise=true , address=64.49.13.137:4500 , piece_id=QJKNHVY5IRULGIELVRDNXQMWI4C6XLPLKH5QX3OLEG4IEOIN4MZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234107016 (51m57.0150404s) [noise=true , address=79.127.219.48:4090 , piece_id=SMYUBZNSBP4OWQZFEIZ5GIZ6QRYLEKXJI5EFDKHSZJID744JTTDA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234225439 (51m57.015158817s) [noise=true , address=79.127.219.51:4210 , piece_id=N5X4TBNI5MBOJKBING7YL55LVQU6B5ZQBSGAM3YQFTVACO4BSC5Q, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234530128 (51m57.01546354s) [noise=true , address=104.36.82.10:4101 , piece_id=S6RT7PW6TVY7K2KNB42YKXEQMDTSMOFJK4CDNO3MKB6FDXLVYBXA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234677395 (51m57.015610776s) [noise=true , address=77.90.31.18:4816 , piece_id=HY55JOE2FS3B3EGYJ3C43MTAXQ33EBQKUUX27SPJ4IMRFVDDOOZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235151053 (51m57.016084431s) [noise=true , address=79.127.220.101:4048 , piece_id=UUV32R7XFPCNDEF43ZOMWC67JJZ7U7YDMLXFXAHTIXUKESNCB4MA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235529599 (51m57.016462976s) [noise=true , address=156.146.43.225:4449 , piece_id=G44WXWCQDQOSSOBG64LRAU3DLOUSH2ULL4I3GT6THFAGYD6BQRTA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235990344 (51m57.016923728s) [noise=true , address=79.127.219.38:4004 , piece_id=JPDFRQ2YEH3KF6RGT4QIE47C6ZI6HMHBTUOXTHGRYZ4ZNDMFEZJQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.236414748 (51m57.017348125s) [noise=true , address=79.127.205.246:4282 , piece_id=HC27CTDAG7TRJ5YQQFFMG7SFDRIN3JRTMFUI42KMK35T3TIB4TZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.237577245 (51m57.018510629s) [noise=true , address=104.36.82.7:4918 , piece_id=3KJZTIAZBP7WBZMHYTYYO2XWUBANMLBT7KUCUKTYHSJ2MZRKXQAA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.238217941 (51m57.019151322s) [noise=true , address=79.127.205.233:4633 , piece_id=VZE6UZ4SW2XBZRNZMBP4TTDDVBRQGYM6SQKYR5BPYBV446K5KPGA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.238950587 (51m57.019883971s) [noise=true , address=77.90.31.11:4571 , piece_id=5XMT4IU4WYZNCLUHFKE2ZYDJTGIX6ESHWWHTNOW22IUB3GE5V3JQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.239460727 (51m57.020394109s) [noise=true , address=77.90.31.10:4538 , piece_id=Z6GPO6B4JMM27B5CJE64GZMAEDWAJKAAVJBRCPRAB5ULHMI7XLFQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.239899361 (51m57.020832748s) [noise=true , address=79.127.205.234:4616 , piece_id=XQSAFSV4ZC5ACSFVVWKAL7BPLQ2KVT2ZEYU6LUM3Q3EFOP3S22PQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.304182241 (51m57.085115671s) [upload=802]: Next returned nil segment Jul 26 22:43:41.304206569 (51m57.085139943s) [upload=802]: Waiting for error group managing segments... Jul 26 22:43:41.309312769 (51m57.090246201s) [noise=true , address=79.127.219.38:4004 , piece_id=JPDFRQ2YEH3KF6RGT4QIE47C6ZI6HMHBTUOXTHGRYZ4ZNDMFEZJQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.366028377 (51m57.146961744s) [noise=true , address=77.90.31.11:4571 , piece_id=5XMT4IU4WYZNCLUHFKE2ZYDJTGIX6ESHWWHTNOW22IUB3GE5V3JQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.385684427 (51m57.166617806s) [noise=true , address=79.127.219.51:4210 , piece_id=N5X4TBNI5MBOJKBING7YL55LVQU6B5ZQBSGAM3YQFTVACO4BSC5Q, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:41.823755988 (51m57.604689367s) [noise=true , address=79.127.205.246:4282 , piece_id=HC27CTDAG7TRJ5YQQFFMG7SFDRIN3JRTMFUI42KMK35T3TIB4TZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.837645590 (51m57.61857896s) [noise=true , address=79.127.205.245:4228 , piece_id=IIVPND2UAM7VD6DA4HGKOX3B3ZVHNYP63ZPWNMN3F47UVZ57TD6A, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.944332236 (51m57.725265612s) [noise=true , address=64.49.13.138:4482 , piece_id=HTUBWJUH7EUYOMFWBLQX25WAJGRJCH3DGX2V6HAZSJ2IUMEI3LCQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:42.082424997 (51m57.86335838s) [noise=true , address=79.127.205.234:4616 , piece_id=XQSAFSV4ZC5ACSFVVWKAL7BPLQ2KVT2ZEYU6LUM3Q3EFOP3S22PQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:42.339902702 (51m58.120836064s) [noise=true , address=79.127.205.233:4633 , piece_id=VZE6UZ4SW2XBZRNZMBP4TTDDVBRQGYM6SQKYR5BPYBV446K5KPGA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:42.616942028 (51m58.397875405s) [noise=true , address=79.127.219.48:4090 , piece_id=SMYUBZNSBP4OWQZFEIZ5GIZ6QRYLEKXJI5EFDKHSZJID744JTTDA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:43.761718655 (51m59.542652024s) [noise=true , address=156.146.43.225:4449 , piece_id=G44WXWCQDQOSSOBG64LRAU3DLOUSH2ULL4I3GT6THFAGYD6BQRTA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:44.144083824 (51m59.925017201s) [noise=true , address=64.49.13.137:4500 , piece_id=QJKNHVY5IRULGIELVRDNXQMWI4C6XLPLKH5QX3OLEG4IEOIN4MZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:45.274140250 (52m1.055073612s) [noise=true , address=79.127.220.101:4048 , piece_id=UUV32R7XFPCNDEF43ZOMWC67JJZ7U7YDMLXFXAHTIXUKESNCB4MA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:46.921015676 (52m2.70194904s) [noise=true , address=77.90.31.19:4874 , piece_id=RLVLQGUYAJRH53SBRIA25EE6L3YCXCHIN7CQ22MGFE5QJZECJYNQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:52.019885342 (52m7.800818718s) [noise=true , address=77.90.31.10:4538 , piece_id=Z6GPO6B4JMM27B5CJE64GZMAEDWAJKAAVJBRCPRAB5ULHMI7XLFQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:59.877267403 (52m15.658200785s) [noise=true , address=77.90.31.18:4816 , piece_id=HY55JOE2FS3B3EGYJ3C43MTAXQ33EBQKUUX27SPJ4IMRFVDDOOZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:44:41.166057950 (52m56.946991332s) [noise=true , address=104.36.82.35:4607 , piece_id=X36Y42PQULA5A4B6UZMCPQK34FM4INNMUMQHKKZL2FG6MOELOJFA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1Ex8eUvL1Hqxd5EffckSvBeZDpGrMGhKENyCPvqHybtQf3YQzM, address:104.36.82.35:4607): protocol: rpc: dial tcp 104.36.82.35:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166171721 (52m56.9471051s) [noise=true , address=104.36.82.10:4101 , piece_id=S6RT7PW6TVY7K2KNB42YKXEQMDTSMOFJK4CDNO3MKB6FDXLVYBXA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166252938 (52m56.947186315s) [noise=true , address=104.36.82.7:4918 , piece_id=3KJZTIAZBP7WBZMHYTYYO2XWUBANMLBT7KUCUKTYHSJ2MZRKXQAA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:12EMGZ4hpXmBp4TwemLrbKAVtjBmtexMzEahRTzFw42qQipsAG, address:104.36.82.7:4918): protocol: write tcp 159.223.143.39:37030->104.36.82.7:4918: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166278974 (52m56.947212365s) [seg_pos={0 0}, upload=802]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:44:41.189117481 (52m56.97005085s) [upload=802]: Done uploading segments... Jul 26 22:44:41.189141942 (52m56.970075317s) [upload=802]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.025697982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XXX.XXX.X.XX..X..X......X.............X................X...] duration: 467.917602ms FAIL (2025-07-26 22:44:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.959855ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 207.169283ms FAIL (2025-07-26 22:44:20 UTC) orbiter warning: uplink: too many requests Duration: 207.197664ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX..X..XX..X..XX.X....X....X........X.......................] duration: 403.258317ms FAIL (2025-07-26 22:44:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.287104ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.933267332s FAIL (2025-07-26 22:44:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.9332919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XX] duration: 1m0.026927028s FAIL (2025-07-26 22:43:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== closed: op Jul 26 22:42:31.875494050 (50m47.656427425s) [seg_pos={0 0}, upload=770]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:42:31.897672595 (50m47.678605975s) [upload=770]: Done uploading segments... Jul 26 22:42:31.897692998 (50m47.678626381s) [upload=770]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Jul 26 22:42:41.119669262 (50m56.90060265s) [upload=788]: Starting upload Jul 26 22:42:41.119746547 (50m56.900679932s) [upload=788]: Done starting upload Jul 26 22:42:41.119970071 (50m56.900903449s) [upload=788]: Uploading segments... Jul 26 22:42:41.120491867 (50m56.901425243s) [upload=788]: Got next segment. Inline: false Jul 26 22:42:41.120566591 (50m56.901499972s) [upload=788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:42:41.120576258 (50m56.901509636s) [upload=788]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:42:41.159698708 (50m56.940632096s) [seg_pos={0 0}, upload=788]: Begin upload segment... Jul 26 22:42:41.159778821 (50m56.940712184s) [seg_pos={0 0}, upload=788]: Done begin upload segment. Jul 26 22:42:41.159819902 (50m56.940753288s) [noise=true , address=79.127.219.38:4004 , piece_id=5LNGIYND5433E5Y6RIXUN5SJ2KZBHAL4VKNM2BETGAJI4ZDRNZJA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.160357367 (50m56.941290743s) [noise=true , address=45.91.116.5:4310 , piece_id=24PKHLS7LIWT5FSSOK6GTAGJA2QT23FGZBTYKCFRRQZIAFUOLJRQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.160740977 (50m56.941674363s) [noise=true , address=77.90.31.21:4900 , piece_id=BPIJVCVJFO2VULVEZCVXKZZJYK4T3ZAQR5SDR2YUED3WVGIHNRBA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.161159365 (50m56.942092751s) [noise=true , address=79.127.219.48:4090 , piece_id=N4YY2MEHQ5GDG5IEUHRZUYCFZEFKWWB6Z746X2BZ3DT53NDECIHA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.161585900 (50m56.942519287s) [noise=true , address=77.90.31.11:4560 , piece_id=2G27HVSVC2D7VMTF7UEKQ46CQOVXOWFYKV6G7R6RQNRJUKHY5O6Q, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.161900536 (50m56.942833901s) [noise=true , address=45.91.116.11:4124 , piece_id=KK2HXQVYAS2WPWBPO3RDXSJH5AJMJIHI45EKAJ4JXMSMYGUOOXWA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.162174785 (50m56.943108149s) [noise=true , address=104.36.82.52:4091 , piece_id=OPAS4F56AZ6MTZG7IGHHHPT3EHFSZKMV6R4BMS4WI2YGBDWAFTHA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.162504222 (50m56.943437606s) [noise=true , address=77.90.31.15:4715 , piece_id=VTL7LRR2FDDRNJMNG5SZDCTLL6NAUAEII52JPV2KSXRDANZ3LWIQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.162815008 (50m56.943748392s) [noise=true , address=79.127.201.215:4347 , piece_id=2YL6SRDVNRLUVTY52NPSD45CSI6GWLRVGFWA3CJO5AOU2INO7EBQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.163231431 (50m56.944164795s) [noise=true , address=79.127.205.233:4649 , piece_id=7L55UVAVWQFODXIYC6QBFLLVDSD6RZBXVAGJ2KGKTBVMPLYRIXCQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.163553660 (50m56.944487037s) [noise=true , address=77.90.31.16:4743 , piece_id=FJ6A2JBU2QRMBBR3S3X6PODEXDDVQ4K534JZYDPAHK2TBVLOO2GA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.163902378 (50m56.944835765s) [noise=true , address=79.127.205.234:4603 , piece_id=AHNYCS34ZHS7OYZGUNE4GRA2UYBZOTSG6BWN37RROM2AFHKTGEXQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.164241933 (50m56.945175313s) [noise=true , address=79.127.220.101:4048 , piece_id=FXAMGR6YV5ROTM4XXV4HIXLSLB6IUJDF5NBABCLGES6IWI6RNJXQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.164543860 (50m56.945477247s) [noise=true , address=77.90.31.18:4809 , piece_id=ZOCAFE6L3S252W7D6DJG7AO2AJWQMB6VDQLGZ5DE2AU45XMAYBAA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.164898196 (50m56.945831571s) [noise=true , address=156.146.43.225:4448 , piece_id=NICLKW2I7FRRO74H2RTUDJNR2XCONV74OAJDEZX52WQPS4PG2VFQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.165214086 (50m56.946147453s) [noise=true , address=104.36.82.44:4877 , piece_id=45N5BVVXQNCYEQTWEY5UE26PMYX34EZTXAU4IZZTUTMUNR7FFYRQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.165481585 (50m56.94641495s) [noise=true , address=79.127.205.245:4243 , piece_id=DIWMJGHBI4X3UH6VZNFOBET3FC3GN5JZXMTEQKIXDDNG5HPA7MQA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.165794133 (50m56.946727514s) [noise=true , address=45.91.116.10:4100 , piece_id=R2OMVYXOKMSAKXYLG36Z5VBRZQJ47ZXS5EXWOMBUT37JJVX4LMEQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.242046962 (50m57.022980346s) [noise=true , address=77.90.31.18:4809 , piece_id=ZOCAFE6L3S252W7D6DJG7AO2AJWQMB6VDQLGZ5DE2AU45XMAYBAA, seg_pos={0 0}, upload=788]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:41.280902777 (50m57.061836155s) [upload=788]: Next returned nil segment Jul 26 22:42:41.280933957 (50m57.061867335s) [upload=788]: Waiting for error group managing segments... Jul 26 22:42:41.297380287 (50m57.078313659s) [noise=true , address=104.36.82.52:4091 , piece_id=OPAS4F56AZ6MTZG7IGHHHPT3EHFSZKMV6R4BMS4WI2YGBDWAFTHA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.310454552 (50m57.091396903s) [noise=true , address=77.90.31.21:4900 , piece_id=BPIJVCVJFO2VULVEZCVXKZZJYK4T3ZAQR5SDR2YUED3WVGIHNRBA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.316342461 (50m57.097275827s) [noise=true , address=77.90.31.11:4560 , piece_id=2G27HVSVC2D7VMTF7UEKQ46CQOVXOWFYKV6G7R6RQNRJUKHY5O6Q, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.519159489 (50m57.300092855s) [noise=true , address=79.127.219.38:4004 , piece_id=5LNGIYND5433E5Y6RIXUN5SJ2KZBHAL4VKNM2BETGAJI4ZDRNZJA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.803436354 (50m57.584369716s) [noise=true , address=79.127.205.233:4649 , piece_id=7L55UVAVWQFODXIYC6QBFLLVDSD6RZBXVAGJ2KGKTBVMPLYRIXCQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.829993125 (50m57.610926504s) [noise=true , address=79.127.205.234:4603 , piece_id=AHNYCS34ZHS7OYZGUNE4GRA2UYBZOTSG6BWN37RROM2AFHKTGEXQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.844040237 (50m57.624973606s) [noise=true , address=79.127.205.245:4243 , piece_id=DIWMJGHBI4X3UH6VZNFOBET3FC3GN5JZXMTEQKIXDDNG5HPA7MQA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.854421567 (50m57.635355015s) [noise=true , address=79.127.219.48:4090 , piece_id=N4YY2MEHQ5GDG5IEUHRZUYCFZEFKWWB6Z746X2BZ3DT53NDECIHA, seg_pos={0 0}, upload=788]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:42.923305857 (50m58.704239235s) [noise=true , address=79.127.220.101:4048 , piece_id=FXAMGR6YV5ROTM4XXV4HIXLSLB6IUJDF5NBABCLGES6IWI6RNJXQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:43.143184591 (50m58.924117958s) [noise=true , address=45.91.116.11:4124 , piece_id=KK2HXQVYAS2WPWBPO3RDXSJH5AJMJIHI45EKAJ4JXMSMYGUOOXWA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:44.504316917 (51m0.285250347s) [noise=true , address=156.146.43.225:4448 , piece_id=NICLKW2I7FRRO74H2RTUDJNR2XCONV74OAJDEZX52WQPS4PG2VFQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:52.128453795 (51m7.909387163s) [noise=true , address=45.91.116.5:4310 , piece_id=24PKHLS7LIWT5FSSOK6GTAGJA2QT23FGZBTYKCFRRQZIAFUOLJRQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:53.288301846 (51m9.069235229s) [noise=true , address=77.90.31.16:4743 , piece_id=FJ6A2JBU2QRMBBR3S3X6PODEXDDVQ4K534JZYDPAHK2TBVLOO2GA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:57.609070108 (51m13.390003477s) [noise=true , address=45.91.116.10:4100 , piece_id=R2OMVYXOKMSAKXYLG36Z5VBRZQJ47ZXS5EXWOMBUT37JJVX4LMEQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:43:06.108143557 (51m21.889076935s) [noise=true , address=79.127.201.215:4347 , piece_id=2YL6SRDVNRLUVTY52NPSD45CSI6GWLRVGFWA3CJO5AOU2INO7EBQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:43:41.085338392 (51m56.866271767s) [noise=true , address=77.90.31.15:4715 , piece_id=VTL7LRR2FDDRNJMNG5SZDCTLL6NAUAEII52JPV2KSXRDANZ3LWIQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:43:41.085668838 (51m56.866602214s) [noise=true , address=104.36.82.44:4877 , piece_id=45N5BVVXQNCYEQTWEY5UE26PMYX34EZTXAU4IZZTUTMUNR7FFYRQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: ecclient: upload failed (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH, address:104.36.82.44:4877): protocol: rpc: dial tcp 104.36.82.44:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:41.085695001 (51m56.866628379s) [seg_pos={0 0}, upload=788]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:43:41.110295795 (51m56.891229164s) [upload=788]: Done uploading segments... Jul 26 22:43:41.110322252 (51m56.891255627s) [upload=788]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.026947669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 204.802569ms FAIL (2025-07-26 22:43:36 UTC) orbiter warning: uplink: too many requests Duration: 204.831684ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.935422751s FAIL (2025-07-26 22:43:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.935445078s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX..X..XX..X..XX.X....X....X........X......................] duration: 420.643964ms FAIL (2025-07-26 22:43:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 420.672891ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XXX.XXX.X.XX..X..X......X.............X................X..] duration: 1.123526498s FAIL (2025-07-26 22:43:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.123559369s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........................X........X.X.X...XX.................] duration: 9.923288844s FAIL (2025-07-26 22:43:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.923314902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 474.242069ms FAIL (2025-07-26 22:43:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.269197ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 191.482832ms FAIL (2025-07-26 22:42:47 UTC) orbiter warning: uplink: too many requests Duration: 191.517994ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.X] duration: 1m0.025796091s FAIL (2025-07-26 22:42:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== m_ObjectCommit Jul 26 22:41:16.892795965 (49m32.673729353s) [upload=766]: Done uploading segments... Jul 26 22:41:16.892822285 (49m32.673755654s) [upload=766]: Upload finished. err: <nil> Jul 26 22:41:31.904533772 (49m47.685467151s) [upload=770]: Starting upload Jul 26 22:41:31.904610378 (49m47.685543748s) [upload=770]: Done starting upload Jul 26 22:41:31.904903375 (49m47.685836742s) [upload=770]: Uploading segments... Jul 26 22:41:31.904947752 (49m47.685881138s) [upload=770]: Got next segment. Inline: false Jul 26 22:41:31.904987837 (49m47.685921213s) [upload=770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:41:31.904993788 (49m47.685927164s) [upload=770]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:41:31.939979572 (49m47.720912958s) [seg_pos={0 0}, upload=770]: Begin upload segment... Jul 26 22:41:31.940234217 (49m47.721167606s) [seg_pos={0 0}, upload=770]: Done begin upload segment. Jul 26 22:41:31.940340475 (49m47.721273864s) [noise=true , address=64.49.13.155:4918 , piece_id=CJC2ZI433LVDE62IBMYGVMUUITLHAFAODXKU6HCUYXKFOS2VRWYA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.940369356 (49m47.721302744s) [noise=true , address=79.127.205.233:4652 , piece_id=SGTPOM6MBVONXYTINHWRXQZDICFXP63DBIN3RC6C7PX7YEV2UWAA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.940917768 (49m47.721851156s) [noise=true , address=64.49.13.140:4508 , piece_id=H67IHSWU6PIPBYYXFPW6JA2XA5WFR6YTJAULCCDYOIFJD7CQPMUQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.941014729 (49m47.721948115s) [noise=true , address=79.127.219.38:4015 , piece_id=ZBU5IXOKKTUA32R3SKRSUZLIN72A3BQM5CSZR3UKMFEG24CSPX3Q, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.941397015 (49m47.722330402s) [noise=true , address=79.127.201.215:4349 , piece_id=LHH6RGB6Z2IS2GAJLVMSJY2VINPH525LJ2XQVFYU6U4B2FMRMCZQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.941616501 (49m47.722549889s) [noise=true , address=79.127.219.47:4061 , piece_id=O5Z43F7YCUWOC3M6N5Q4AOBHOCVYDU4TXYGFBKF3CA4EQS6O4XHQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.941841662 (49m47.722775049s) [noise=true , address=45.91.116.9:4269 , piece_id=6W4AHYW2RTYCJFLUWWEQVMXIAAXMG4UPQZFLHIKH7BUJOLVGFZWQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.942214512 (49m47.723147899s) [noise=true , address=64.49.13.138:4483 , piece_id=TLODN3U5S237CIEQ2DTAFCCVATLQPSOZIT6I7LE3QTQON44FSMGQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.942250992 (49m47.723184378s) [noise=true , address=77.90.31.12:4601 , piece_id=BVGAHU3DT4N6WSQDH6OMZMPGZZAN7FUZJIUSQV5OAWL7UKHPPVVA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.942698123 (49m47.723631511s) [noise=true , address=77.90.31.10:4545 , piece_id=U2RLVAI42C4H7DWORUHEHTQHDMRC4VBTIVPGQM32GSRC32YPWNOQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.942724421 (49m47.723657807s) [noise=true , address=77.90.31.11:4571 , piece_id=YGROPLWOPWHEP63LQNHLRFXBTCT242IEBD7QS3INRZDLAGIHOSYA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.943115112 (49m47.724048499s) [noise=true , address=77.90.31.13:4659 , piece_id=3FLC6YAKNVCOEIKINJONB6ZOEFWKSQDY43WGOROWP3LW5CTQVWJQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.943267669 (49m47.724201054s) [noise=true , address=156.146.43.225:4448 , piece_id=ROPLKCB3V7NOI6I7PAUV6V3CV44LYY2XM2S4N4GMX2NW55HIPGXA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.943676721 (49m47.724610108s) [noise=true , address=64.49.13.170:4719 , piece_id=FF5VCNAKKEZDNK52HLYV3AN72SX4DA3WWUIWRDUM7ZZXC7VJP7JQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.944119065 (49m47.725052454s) [noise=true , address=79.127.205.234:4591 , piece_id=BIVT2MMYNKGMZ3AUUZARNE3PTYBUDFPY4FCXCESHZHM6LZHYFBEQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.944536652 (49m47.72547004s) [noise=true , address=79.127.205.246:4273 , piece_id=ETVT4DT33BC5FKBW4UBSLVYUHIKPEJZ2O2IAAQWNXAWRYIBFUSSQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.946399497 (49m47.727332882s) [noise=true , address=79.127.219.48:4090 , piece_id=EYKSSYL5KBMAEX56CIS6QSB4MLJLBWYYIA25P45BCGCR47VFGASQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.946962143 (49m47.72789553s) [noise=true , address=45.91.116.5:4310 , piece_id=7KIIN5J4UGRI4EZPNLQC262CCGXBZCR6GWKVIH2CL5EXHDUMJJOA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.992845394 (49m47.77377876s) [noise=true , address=79.127.219.47:4061 , piece_id=O5Z43F7YCUWOC3M6N5Q4AOBHOCVYDU4TXYGFBKF3CA4EQS6O4XHQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:32.065203529 (49m47.846136918s) [upload=770]: Next returned nil segment Jul 26 22:41:32.065227477 (49m47.846160866s) [upload=770]: Waiting for error group managing segments... Jul 26 22:41:32.071542785 (49m47.852476163s) [noise=true , address=79.127.219.38:4015 , piece_id=ZBU5IXOKKTUA32R3SKRSUZLIN72A3BQM5CSZR3UKMFEG24CSPX3Q, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:32.089074327 (49m47.870007692s) [noise=true , address=77.90.31.10:4545 , piece_id=U2RLVAI42C4H7DWORUHEHTQHDMRC4VBTIVPGQM32GSRC32YPWNOQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:32.092848526 (49m47.873781893s) [noise=true , address=156.146.43.225:4448 , piece_id=ROPLKCB3V7NOI6I7PAUV6V3CV44LYY2XM2S4N4GMX2NW55HIPGXA, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:32.131668532 (49m47.912601913s) [noise=true , address=77.90.31.11:4571 , piece_id=YGROPLWOPWHEP63LQNHLRFXBTCT242IEBD7QS3INRZDLAGIHOSYA, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:32.279396058 (49m48.060329441s) [noise=true , address=79.127.219.48:4090 , piece_id=EYKSSYL5KBMAEX56CIS6QSB4MLJLBWYYIA25P45BCGCR47VFGASQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:32.305220249 (49m48.086153626s) [noise=true , address=64.49.13.155:4918 , piece_id=CJC2ZI433LVDE62IBMYGVMUUITLHAFAODXKU6HCUYXKFOS2VRWYA, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:1YjPvRvhJkNJtmt5CJ7nUst9NicEGWzpe7Pn1GWes5uc9H1gTK, address:64.49.13.155:4918): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:32.518001703 (49m48.29893507s) [noise=true , address=79.127.205.234:4591 , piece_id=BIVT2MMYNKGMZ3AUUZARNE3PTYBUDFPY4FCXCESHZHM6LZHYFBEQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:32.553910144 (49m48.334843522s) [noise=true , address=79.127.205.233:4652 , piece_id=SGTPOM6MBVONXYTINHWRXQZDICFXP63DBIN3RC6C7PX7YEV2UWAA, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:32.779618896 (49m48.560552265s) [noise=true , address=79.127.205.246:4273 , piece_id=ETVT4DT33BC5FKBW4UBSLVYUHIKPEJZ2O2IAAQWNXAWRYIBFUSSQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:33.359278311 (49m49.140211675s) [noise=true , address=45.91.116.9:4269 , piece_id=6W4AHYW2RTYCJFLUWWEQVMXIAAXMG4UPQZFLHIKH7BUJOLVGFZWQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:34.554009667 (49m50.334943054s) [noise=true , address=45.91.116.5:4310 , piece_id=7KIIN5J4UGRI4EZPNLQC262CCGXBZCR6GWKVIH2CL5EXHDUMJJOA, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:37.333217000 (49m53.114150378s) [noise=true , address=64.49.13.170:4719 , piece_id=FF5VCNAKKEZDNK52HLYV3AN72SX4DA3WWUIWRDUM7ZZXC7VJP7JQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:38.782141186 (49m54.563074564s) [noise=true , address=64.49.13.140:4508 , piece_id=H67IHSWU6PIPBYYXFPW6JA2XA5WFR6YTJAULCCDYOIFJD7CQPMUQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:39.873142320 (49m55.654075689s) [noise=true , address=64.49.13.138:4483 , piece_id=TLODN3U5S237CIEQ2DTAFCCVATLQPSOZIT6I7LE3QTQON44FSMGQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:54.209822084 (50m9.990755464s) [noise=true , address=79.127.201.215:4349 , piece_id=LHH6RGB6Z2IS2GAJLVMSJY2VINPH525LJ2XQVFYU6U4B2FMRMCZQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:42:02.361295715 (50m18.142229099s) [noise=true , address=77.90.31.12:4601 , piece_id=BVGAHU3DT4N6WSQDH6OMZMPGZZAN7FUZJIUSQV5OAWL7UKHPPVVA, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:42:31.875460070 (50m47.656393436s) [noise=true , address=77.90.31.13:4659 , piece_id=3FLC6YAKNVCOEIKINJONB6ZOEFWKSQDY43WGOROWP3LW5CTQVWJQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: write tcp 159.223.143.39:55040->77.90.31.13:4659: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:42:31.875494050 (50m47.656427425s) [seg_pos={0 0}, upload=770]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:42:31.897672595 (50m47.678605975s) [upload=770]: Done uploading segments... Jul 26 22:42:31.897692998 (50m47.678626381s) [upload=770]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.025819598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [...XXX.XXX.X.XX..X..X......X.............X................X.] duration: 405.483793ms FAIL (2025-07-26 22:42:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 405.517656ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........................X........X.X.X...XX................] duration: 13.924944109s FAIL (2025-07-26 22:42:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.924971217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXX..X..XX..X..XX.X....X....X........X.....................] duration: 583.470676ms FAIL (2025-07-26 22:42:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 583.516144ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 673.218567ms FAIL (2025-07-26 22:42:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 673.25958ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........................X........X.X.X...XX...............] duration: 44.361372717s FAIL (2025-07-26 22:42:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.361424826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [....XXX.XXX.X.XX..X..X......X.............X................X] duration: 30.033467865s FAIL (2025-07-26 22:41:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.033498193s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XXX..X..XX..X..XX.X....X....X........X....................] duration: 470.064914ms FAIL (2025-07-26 22:41:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.106652ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 235.749699ms FAIL (2025-07-26 22:41:47 UTC) orbiter warning: uplink: too many requests Duration: 235.778412ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...........................X........X.X.X...XX..............] duration: 25.962093869s FAIL (2025-07-26 22:41:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.962126387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 527.782355ms FAIL (2025-07-26 22:41:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 527.81373ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.] duration: 7.577289004s FAIL (2025-07-26 22:41:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 26 22:40:08.910912462 (48m24.691845855s) [upload=733]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Jul 26 22:40:09.915432325 (48m25.696365696s) [upload=751]: Starting upload Jul 26 22:40:09.915584484 (48m25.696517863s) [upload=751]: Done starting upload Jul 26 22:40:09.915753736 (48m25.696687122s) [upload=751]: Uploading segments... Jul 26 22:40:09.916091771 (48m25.697025143s) [upload=751]: Got next segment. Inline: false Jul 26 22:40:09.916255900 (48m25.69718928s) [upload=751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:40:09.916265701 (48m25.697199087s) [upload=751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:40:09.951897011 (48m25.732830395s) [seg_pos={0 0}, upload=751]: Begin upload segment... Jul 26 22:40:09.951971898 (48m25.732905275s) [seg_pos={0 0}, upload=751]: Done begin upload segment. Jul 26 22:40:09.952009426 (48m25.732942831s) [noise=true , address=79.127.219.47:4048 , piece_id=3XLKJ3TFG5VPRH72DHQVY43Z4YCPQTIEMIIZ5OECNPT3GBYU2JBQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952301126 (48m25.73323451s) [noise=true , address=193.149.164.143:4702 , piece_id=3B5Q4ZNGKLNR3KR462B2M5RJ6Q5MV4YMHKE4TEKDP7BK7SLHTV6Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952758802 (48m25.733692182s) [noise=true , address=104.36.82.11:4128 , piece_id=VMYPCTBSSZAGY6LJ37SMGOZ4JG3LF4ZY37FNUMZ7SPIKJYCOD2XA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952918304 (48m25.733851684s) [noise=true , address=79.127.201.215:4346 , piece_id=2J4IPB3KOKYNZND6M7UM2Z6N6KQEDFICFUGG6VITRLOFXGFUX5YQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953453113 (48m25.734386505s) [noise=true , address=104.36.82.11:4126 , piece_id=NRXAD3XG2NTYTF34MS6COECYG5MAYZB3SCGRGO3GUQARKPWMDQYA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953499878 (48m25.734433267s) [noise=true , address=64.49.13.170:4718 , piece_id=H7FHQNRUITABERXO25B4VIFMA7MHI7WITF35M5Z2W5X2J3MMRWBA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953960495 (48m25.734893878s) [noise=true , address=79.127.205.246:4284 , piece_id=MF2AMEP4HEYFDPNR56T334XLNPCBCEDMUX5A6M4NGFLFQATYNDPQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954069989 (48m25.735003423s) [noise=true , address=79.127.205.234:4615 , piece_id=OMMVZY5FYCWXJWAOGGNZLGSBBK6EDXAOF5AKGNQJUR5HTBRQP5AA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954610643 (48m25.73554403s) [noise=true , address=79.127.205.233:4649 , piece_id=AJ3GJHEJVOYRDV2MDNCKRI6RGAWPCYEWFOXUBVZKWX6EATNW6MNQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954666256 (48m25.735599648s) [noise=true , address=156.146.43.225:4449 , piece_id=CAU3I4RIWPVL7A3BUYMOAOTUMUTM7VS5UMJSAXKQD3U3POTTSCAA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955031508 (48m25.735964894s) [noise=true , address=79.127.219.51:4210 , piece_id=PUGY4EABLOOS7JI3G4VWAJG3FWQNVKIQN6F33FBT23KF7XCAQMAQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955294354 (48m25.736227735s) [noise=true , address=79.127.205.245:4252 , piece_id=4PTGQYPDPYRSXVWWBIOYEGGHCURNHIERME4N3G3YIGOM5LB5MVPA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955495328 (48m25.736428714s) [noise=true , address=79.127.205.235:4689 , piece_id=3GYWKGWSJVVIRIYA7MMHIQ2ICPPM6SXBQ5WBT7B5HVTT4UWANN6Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955851690 (48m25.736785076s) [noise=true , address=77.90.31.17:4800 , piece_id=4ZHOMLKPLJ2AQJA7GKFY3CWA53UVIUD6MZVWGGFC7E35DY5SSA7Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955925977 (48m25.73685936s) [noise=true , address=45.91.116.14:4222 , piece_id=QPG5O763WPESAJS7EPW7QKZBOAO3ELEFRPUD6DYTETVISSMQMPAA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.956391693 (48m25.737325078s) [noise=true , address=104.36.82.2:4755 , piece_id=4MESSBGPPVRO6IQIZUFG4J6PRFPPKCH3AXIJVPFPOZZ3XRMNTRWA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.956928609 (48m25.737861989s) [noise=true , address=79.127.219.48:4075 , piece_id=TSPDPWCCWO6L3ZKO26NU3RXVZK6J7WNF2F3MCRRRUK2JQFMSIBDA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.957437578 (48m25.738370965s) [noise=true , address=77.90.31.14:4684 , piece_id=IU4OAXNVIMFE45KML4GUMQDRHEZSKGTJPK5DEWSEOMBWYHJMDIYQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:10.010392040 (48m25.791325428s) [noise=true , address=79.127.219.51:4210 , piece_id=PUGY4EABLOOS7JI3G4VWAJG3FWQNVKIQN6F33FBT23KF7XCAQMAQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:10.075537395 (48m25.856470757s) [upload=751]: Next returned nil segment Jul 26 22:40:10.075558705 (48m25.856492082s) [upload=751]: Waiting for error group managing segments... Jul 26 22:40:10.126404804 (48m25.907338204s) [noise=true , address=77.90.31.17:4800 , piece_id=4ZHOMLKPLJ2AQJA7GKFY3CWA53UVIUD6MZVWGGFC7E35DY5SSA7Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.579064023 (48m26.359997387s) [noise=true , address=79.127.205.246:4284 , piece_id=MF2AMEP4HEYFDPNR56T334XLNPCBCEDMUX5A6M4NGFLFQATYNDPQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.592223660 (48m26.373157044s) [noise=true , address=79.127.205.235:4689 , piece_id=3GYWKGWSJVVIRIYA7MMHIQ2ICPPM6SXBQ5WBT7B5HVTT4UWANN6Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.625813526 (48m26.406746892s) [noise=true , address=79.127.205.234:4615 , piece_id=OMMVZY5FYCWXJWAOGGNZLGSBBK6EDXAOF5AKGNQJUR5HTBRQP5AA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.860506119 (48m26.641439485s) [noise=true , address=79.127.205.245:4252 , piece_id=4PTGQYPDPYRSXVWWBIOYEGGHCURNHIERME4N3G3YIGOM5LB5MVPA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.873202746 (48m26.654136111s) [noise=true , address=79.127.205.233:4649 , piece_id=AJ3GJHEJVOYRDV2MDNCKRI6RGAWPCYEWFOXUBVZKWX6EATNW6MNQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:11.115326185 (48m26.896259588s) [noise=true , address=193.149.164.143:4702 , piece_id=3B5Q4ZNGKLNR3KR462B2M5RJ6Q5MV4YMHKE4TEKDP7BK7SLHTV6Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:11.717107067 (48m27.498040445s) [noise=true , address=79.127.219.48:4075 , piece_id=TSPDPWCCWO6L3ZKO26NU3RXVZK6J7WNF2F3MCRRRUK2JQFMSIBDA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:12.148650635 (48m27.929584023s) [noise=true , address=79.127.219.47:4048 , piece_id=3XLKJ3TFG5VPRH72DHQVY43Z4YCPQTIEMIIZ5OECNPT3GBYU2JBQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:13.464654731 (48m29.24558811s) [noise=true , address=64.49.13.170:4718 , piece_id=H7FHQNRUITABERXO25B4VIFMA7MHI7WITF35M5Z2W5X2J3MMRWBA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:14.660571356 (48m30.441504723s) [noise=true , address=156.146.43.225:4449 , piece_id=CAU3I4RIWPVL7A3BUYMOAOTUMUTM7VS5UMJSAXKQD3U3POTTSCAA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:24.682119347 (48m40.463052728s) [noise=true , address=77.90.31.14:4684 , piece_id=IU4OAXNVIMFE45KML4GUMQDRHEZSKGTJPK5DEWSEOMBWYHJMDIYQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:25.404814950 (48m41.185748318s) [noise=true , address=45.91.116.14:4222 , piece_id=QPG5O763WPESAJS7EPW7QKZBOAO3ELEFRPUD6DYTETVISSMQMPAA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:59.474246858 (49m15.255180221s) [noise=true , address=79.127.201.215:4346 , piece_id=2J4IPB3KOKYNZND6M7UM2Z6N6KQEDFICFUGG6VITRLOFXGFUX5YQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:41:09.882265336 (49m25.663198715s) [noise=true , address=104.36.82.11:4128 , piece_id=VMYPCTBSSZAGY6LJ37SMGOZ4JG3LF4ZY37FNUMZ7SPIKJYCOD2XA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS, address:104.36.82.11:4128): protocol: write tcp 159.223.143.39:60630->104.36.82.11:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882373704 (49m25.663307086s) [noise=true , address=104.36.82.11:4126 , piece_id=NRXAD3XG2NTYTF34MS6COECYG5MAYZB3SCGRGO3GUQARKPWMDQYA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:12R3SVXrhJsmYTXR4vDSkfaiqDRox21u17SLmQc9sEab1B7zZRr, address:104.36.82.11:4126): protocol: write tcp 159.223.143.39:51650->104.36.82.11:4126: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882429118 (49m25.663362498s) [noise=true , address=104.36.82.2:4755 , piece_id=4MESSBGPPVRO6IQIZUFG4J6PRFPPKCH3AXIJVPFPOZZ3XRMNTRWA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: write tcp 159.223.143.39:34790->104.36.82.2:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882450347 (49m25.663383729s) [seg_pos={0 0}, upload=751]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:41:09.906834186 (49m25.687767566s) [upload=751]: Done uploading segments... Jul 26 22:41:09.906864582 (49m25.687797962s) [upload=751]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.577315388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..XXX..X..XX..X..XX.X....X....X........X...................] duration: 467.803741ms FAIL (2025-07-26 22:41:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.841174ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX] duration: 1m0.038348053s FAIL (2025-07-26 22:41:09 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 26 22:40:08.910912462 (48m24.691845855s) [upload=733]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Jul 26 22:40:09.915432325 (48m25.696365696s) [upload=751]: Starting upload Jul 26 22:40:09.915584484 (48m25.696517863s) [upload=751]: Done starting upload Jul 26 22:40:09.915753736 (48m25.696687122s) [upload=751]: Uploading segments... Jul 26 22:40:09.916091771 (48m25.697025143s) [upload=751]: Got next segment. Inline: false Jul 26 22:40:09.916255900 (48m25.69718928s) [upload=751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:40:09.916265701 (48m25.697199087s) [upload=751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:40:09.951897011 (48m25.732830395s) [seg_pos={0 0}, upload=751]: Begin upload segment... Jul 26 22:40:09.951971898 (48m25.732905275s) [seg_pos={0 0}, upload=751]: Done begin upload segment. Jul 26 22:40:09.952009426 (48m25.732942831s) [noise=true , address=79.127.219.47:4048 , piece_id=3XLKJ3TFG5VPRH72DHQVY43Z4YCPQTIEMIIZ5OECNPT3GBYU2JBQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952301126 (48m25.73323451s) [noise=true , address=193.149.164.143:4702 , piece_id=3B5Q4ZNGKLNR3KR462B2M5RJ6Q5MV4YMHKE4TEKDP7BK7SLHTV6Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952758802 (48m25.733692182s) [noise=true , address=104.36.82.11:4128 , piece_id=VMYPCTBSSZAGY6LJ37SMGOZ4JG3LF4ZY37FNUMZ7SPIKJYCOD2XA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952918304 (48m25.733851684s) [noise=true , address=79.127.201.215:4346 , piece_id=2J4IPB3KOKYNZND6M7UM2Z6N6KQEDFICFUGG6VITRLOFXGFUX5YQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953453113 (48m25.734386505s) [noise=true , address=104.36.82.11:4126 , piece_id=NRXAD3XG2NTYTF34MS6COECYG5MAYZB3SCGRGO3GUQARKPWMDQYA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953499878 (48m25.734433267s) [noise=true , address=64.49.13.170:4718 , piece_id=H7FHQNRUITABERXO25B4VIFMA7MHI7WITF35M5Z2W5X2J3MMRWBA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953960495 (48m25.734893878s) [noise=true , address=79.127.205.246:4284 , piece_id=MF2AMEP4HEYFDPNR56T334XLNPCBCEDMUX5A6M4NGFLFQATYNDPQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954069989 (48m25.735003423s) [noise=true , address=79.127.205.234:4615 , piece_id=OMMVZY5FYCWXJWAOGGNZLGSBBK6EDXAOF5AKGNQJUR5HTBRQP5AA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954610643 (48m25.73554403s) [noise=true , address=79.127.205.233:4649 , piece_id=AJ3GJHEJVOYRDV2MDNCKRI6RGAWPCYEWFOXUBVZKWX6EATNW6MNQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954666256 (48m25.735599648s) [noise=true , address=156.146.43.225:4449 , piece_id=CAU3I4RIWPVL7A3BUYMOAOTUMUTM7VS5UMJSAXKQD3U3POTTSCAA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955031508 (48m25.735964894s) [noise=true , address=79.127.219.51:4210 , piece_id=PUGY4EABLOOS7JI3G4VWAJG3FWQNVKIQN6F33FBT23KF7XCAQMAQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955294354 (48m25.736227735s) [noise=true , address=79.127.205.245:4252 , piece_id=4PTGQYPDPYRSXVWWBIOYEGGHCURNHIERME4N3G3YIGOM5LB5MVPA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955495328 (48m25.736428714s) [noise=true , address=79.127.205.235:4689 , piece_id=3GYWKGWSJVVIRIYA7MMHIQ2ICPPM6SXBQ5WBT7B5HVTT4UWANN6Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955851690 (48m25.736785076s) [noise=true , address=77.90.31.17:4800 , piece_id=4ZHOMLKPLJ2AQJA7GKFY3CWA53UVIUD6MZVWGGFC7E35DY5SSA7Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955925977 (48m25.73685936s) [noise=true , address=45.91.116.14:4222 , piece_id=QPG5O763WPESAJS7EPW7QKZBOAO3ELEFRPUD6DYTETVISSMQMPAA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.956391693 (48m25.737325078s) [noise=true , address=104.36.82.2:4755 , piece_id=4MESSBGPPVRO6IQIZUFG4J6PRFPPKCH3AXIJVPFPOZZ3XRMNTRWA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.956928609 (48m25.737861989s) [noise=true , address=79.127.219.48:4075 , piece_id=TSPDPWCCWO6L3ZKO26NU3RXVZK6J7WNF2F3MCRRRUK2JQFMSIBDA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.957437578 (48m25.738370965s) [noise=true , address=77.90.31.14:4684 , piece_id=IU4OAXNVIMFE45KML4GUMQDRHEZSKGTJPK5DEWSEOMBWYHJMDIYQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:10.010392040 (48m25.791325428s) [noise=true , address=79.127.219.51:4210 , piece_id=PUGY4EABLOOS7JI3G4VWAJG3FWQNVKIQN6F33FBT23KF7XCAQMAQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:10.075537395 (48m25.856470757s) [upload=751]: Next returned nil segment Jul 26 22:40:10.075558705 (48m25.856492082s) [upload=751]: Waiting for error group managing segments... Jul 26 22:40:10.126404804 (48m25.907338204s) [noise=true , address=77.90.31.17:4800 , piece_id=4ZHOMLKPLJ2AQJA7GKFY3CWA53UVIUD6MZVWGGFC7E35DY5SSA7Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.579064023 (48m26.359997387s) [noise=true , address=79.127.205.246:4284 , piece_id=MF2AMEP4HEYFDPNR56T334XLNPCBCEDMUX5A6M4NGFLFQATYNDPQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.592223660 (48m26.373157044s) [noise=true , address=79.127.205.235:4689 , piece_id=3GYWKGWSJVVIRIYA7MMHIQ2ICPPM6SXBQ5WBT7B5HVTT4UWANN6Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.625813526 (48m26.406746892s) [noise=true , address=79.127.205.234:4615 , piece_id=OMMVZY5FYCWXJWAOGGNZLGSBBK6EDXAOF5AKGNQJUR5HTBRQP5AA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.860506119 (48m26.641439485s) [noise=true , address=79.127.205.245:4252 , piece_id=4PTGQYPDPYRSXVWWBIOYEGGHCURNHIERME4N3G3YIGOM5LB5MVPA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.873202746 (48m26.654136111s) [noise=true , address=79.127.205.233:4649 , piece_id=AJ3GJHEJVOYRDV2MDNCKRI6RGAWPCYEWFOXUBVZKWX6EATNW6MNQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:11.115326185 (48m26.896259588s) [noise=true , address=193.149.164.143:4702 , piece_id=3B5Q4ZNGKLNR3KR462B2M5RJ6Q5MV4YMHKE4TEKDP7BK7SLHTV6Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:11.717107067 (48m27.498040445s) [noise=true , address=79.127.219.48:4075 , piece_id=TSPDPWCCWO6L3ZKO26NU3RXVZK6J7WNF2F3MCRRRUK2JQFMSIBDA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:12.148650635 (48m27.929584023s) [noise=true , address=79.127.219.47:4048 , piece_id=3XLKJ3TFG5VPRH72DHQVY43Z4YCPQTIEMIIZ5OECNPT3GBYU2JBQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:13.464654731 (48m29.24558811s) [noise=true , address=64.49.13.170:4718 , piece_id=H7FHQNRUITABERXO25B4VIFMA7MHI7WITF35M5Z2W5X2J3MMRWBA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:14.660571356 (48m30.441504723s) [noise=true , address=156.146.43.225:4449 , piece_id=CAU3I4RIWPVL7A3BUYMOAOTUMUTM7VS5UMJSAXKQD3U3POTTSCAA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:24.682119347 (48m40.463052728s) [noise=true , address=77.90.31.14:4684 , piece_id=IU4OAXNVIMFE45KML4GUMQDRHEZSKGTJPK5DEWSEOMBWYHJMDIYQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:25.404814950 (48m41.185748318s) [noise=true , address=45.91.116.14:4222 , piece_id=QPG5O763WPESAJS7EPW7QKZBOAO3ELEFRPUD6DYTETVISSMQMPAA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:59.474246858 (49m15.255180221s) [noise=true , address=79.127.201.215:4346 , piece_id=2J4IPB3KOKYNZND6M7UM2Z6N6KQEDFICFUGG6VITRLOFXGFUX5YQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:41:09.882265336 (49m25.663198715s) [noise=true , address=104.36.82.11:4128 , piece_id=VMYPCTBSSZAGY6LJ37SMGOZ4JG3LF4ZY37FNUMZ7SPIKJYCOD2XA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS, address:104.36.82.11:4128): protocol: write tcp 159.223.143.39:60630->104.36.82.11:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882373704 (49m25.663307086s) [noise=true , address=104.36.82.11:4126 , piece_id=NRXAD3XG2NTYTF34MS6COECYG5MAYZB3SCGRGO3GUQARKPWMDQYA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:12R3SVXrhJsmYTXR4vDSkfaiqDRox21u17SLmQc9sEab1B7zZRr, address:104.36.82.11:4126): protocol: write tcp 159.223.143.39:51650->104.36.82.11:4126: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882429118 (49m25.663362498s) [noise=true , address=104.36.82.2:4755 , piece_id=4MESSBGPPVRO6IQIZUFG4J6PRFPPKCH3AXIJVPFPOZZ3XRMNTRWA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: write tcp 159.223.143.39:34790->104.36.82.2:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882450347 (49m25.663383729s) [seg_pos={0 0}, upload=751]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:41:09.906834186 (49m25.687767566s) [upload=751]: Done uploading segments... Jul 26 22:41:09.906864582 (49m25.687797962s) [upload=751]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.038381337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX..XXX..X..XX..X..XX.X....X....X........X..................] duration: 542.836713ms FAIL (2025-07-26 22:40:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 542.87346ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)